不能用cx_Freeze制作独立的二进制scrapy蜘蛛

Question

关于我的工作环境的简短描述:win 7 x64,python 2.7 x64,scrapy 0.22,cx_Freeze 4.3.2.

首先,我开发了一个简单的爬行蜘蛛,它工作正常.然后,使用核心scrapy API,我创建了一个外部脚本main.py,它可以运行spider,它也可以根据需要运行.这是脚本的代码:

# external main.py using scrapy core API, 'test' is just replaced name of my project
from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from test.spiders.testSpider import TestSpider
from test import settings, pipelines
from scrapy.utils.project import get_project_settings

spider = TestSpider(domain='test.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()

所以现在我试图让所有的这种二元用setup.py置身于另一个话题cx_Freeze 这里.这是代码:

from cx_Freeze import setup, Executable

includes = ['scrapy', 'pkg_resources', 'lxml.etree', 'lxml._elementpath']

build_options = {'compressed' : True,
                'optimize' : 2,
                'namespace_packages' : ['zope', 'scrapy', 'pkg_resources'],
                'includes' : includes,
                'excludes' : []}

executable = Executable(script='main.py',
                        copyDependentFiles=True,
                        includes=includes)

setup(name='Stand-alone scraper',
      version='0.1',
      description='Stand-alone scraper',
      options= {'build_exe': build_options},
      executables=[executable])

它通常编译成exe文件.我尝试运行它时会出现问题:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in       <module>
    exec code in m.__dict__
  File "main.py", line 2, in <module>
    from scrapy.crawler import Crawler
  File "C:\Python27\lib\site-packages\scrapy\__init__.py", line 6, in <module>
    __version__ = pkgutil.get_data(__package__, 'VERSION').strip()
  File "C:\Python27\lib\pkgutil.py", line 591, in get_data
    return loader.get_data(resource_name)
IOError: [Errno 2] No such file or directory: 'scrapy\\VERSION'

我解决了这个问题,只是将scrapy\version文件从原始源(python\lib\site-packages\scrapy)移动到build-folder中的library.zip\scapy.在第二次运行main.exe后,我收到了另一条消息:

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
    exec code in m.__dict__
  File "main.py", line 11, in <module>
    crawler = Crawler(settings)
  File "C:\Python27\lib\site-packages\scrapy\crawler.py", line 20, in __init__
    self.stats = load_object(settings['STATS_CLASS'])(self)
  File "C:\Python27\lib\site-packages\scrapy\utils\misc.py", line 42, in load_object
    raise ImportError("Error loading object '%s': %s" % (path, e))
ImportError: Error loading object 'scrapy.statscol.MemoryStatsCollector': No module named statscol

我没有找到任何解决方案,只是尝试从我的main.py中的错误消息导入模块.简而言之 - 没有用.每个新的导入我得到一个新的消息与另一个模块(我试图导入15 :))模块,直到在密码学中得到关于aes模块的错误.我也尝试使用像py2exe和pyinstaller这样的cx_freeze替代品,但结果相同.

有人可以帮我解决这个问题吗？感谢您阅读这一点.

Answer 1

用此替换您的 cx_Freeze 代码。

import sys 
    from cx_Freeze import setup, Executable 
    build_exe_options = {"packages": ["os","twisted","scrapy","test"], "excludes": ["tkinter"],"include_msvcr":True} 

    base = None
    setup(  name = "MyScript", 
            version = "0.1",
            description = "Demo", 
            options = {"build_exe": build_exe_options}, 
            executables = [Executable("C:\\MyScript", base=base)]) 

代码的不同之处在于我已经包含了整个包，因此您可以访问它们的所有功能。