标签: twisted.internet

我必须从另一个python文件中调用crawler,我使用以下代码.

def crawl_koovs():
    spider = SomeSpider()
    settings = get_project_settings()
    crawler = Crawler(settings)
    crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
    crawler.configure()
    crawler.crawl(spider)
    crawler.start()
    log.start()
    reactor.run()

在运行它时,我得到错误

exceptions.ValueError: signal only works in main thread

我能找到的唯一解决方法就是使用

reactor.run(installSignalHandlers=False)

我不想使用,因为我想多次调用此方法,并希望在下一次调用之前停止reactor.我可以做些什么来完成这项工作(可能会强制爬虫在相同的'主'线程中启动)？

当扭曲的反应堆正在运行并且在未捕获的延迟内发生异常时,"未处理的错误"将与回溯和异常一起打印到终端.是否可以处理/拦截这些异常(例如,设置回调或覆盖方法)？

编辑:我知道我可以通过向deferrerd添加错误来解决问题.我想知道的是,是否有办法拦截一条未经处理的故障/异常,该故障/异常已经越过链条进入反应堆.

编辑:基本上,我想知道扭曲的反应堆是否有一个全局错误处理程序或可以访问的东西.我想知道因为它打印了故障的追溯和错误.

例:

Unhandled Error
Traceback (most recent call last):
  File "/var/projects/python/server.py", line 359, in run_server
    return server.run()
  File "/var/projects/python/server.py", line 881, in run
    reactor.run()
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.0.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1162, in run
    self.mainLoop()
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.0.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 1171, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/usr/local/lib/python2.6/dist-packages/Twisted-11.0.0-py2.6-linux-x86_64.egg/twisted/internet/base.py", line 793, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/var/projects/python/server.py", line 524, in monitor
    elapsed = time.time() - info.last
exceptions.NameError: global name 'info' is not defined

我一直在尝试用Python创建一个Scrapy具有以下功能的应用程序:

• 一个REST API(我曾提出,用瓶)监听所有的请求抓取/废料和爬行后返回响应.(爬行部分足够短,所以连接可以保持活动,直到爬行得到完成.)

我可以使用以下代码执行此操作:

items = []
def add_item(item):
    items.append(item)

# set up crawler
crawler = Crawler(SpiderClass,settings=get_project_settings())
crawler.signals.connect(add_item, signal=signals.item_passed)

# This is added to make the reactor stop, if I don't use this, the code stucks at reactor.run() line.
crawler.signals.connect(reactor.stop, signal=signals.spider_closed) #@UndefinedVariable 
crawler.crawl(requestParams=requestParams)
# start crawling 
reactor.run() #@UndefinedVariable
return str(items)

现在我面临的问题是在停止反应堆之后(这对我来说似乎是必要的,因为我不想坚持reactor.run()).第一次请求后我无法接受进一步的请求.第一次请求完成后,我收到以下错误:

Traceback (most recent call last):
  File "c:\python27\lib\site-packages\flask\app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e) …

遇到麻烦,服务器程序运行一段时间后,会出现大量错误(错误),服务器只能接受客户端连接但不接受数据,最后暂停动画.以下错误:

//report error:
//report error:
Unhandled Error
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/log.py", line 69, in callWithContext
        return context.call({ILogContext: newCtx}, func, *args, **kw)
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/context.py", line 118, in callWithContext
        return self.currentContext().callWithContext(ctx, func, *args, **kw)
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/python/context.py", line 81, in callWithContext
        return func(*args,**kw)
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
        why = selectable.doRead()
//        --- <exception caught here> ---
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 1016, in doRead
        transport = self.transport(skt, protocol, addr, self, s, self.reactor)
      File "/usr/lib64/python2.7/lib/python2.7/site-packages/Twisted-12.1.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 773, in __init__
        self.startReading()
      File …

我正在尝试从脚本运行 scrapy，但无法让程序创建导出文件

我尝试以两种不同的方式导出文件：

1. 有管道
2. 具有饲料出口。

当我从命令行运行 scrapy 时，这两种方法都有效，但当我从脚本运行 scrapy 时，这两种方法都不起作用。

我并不是唯一遇到这个问题的人。这是另外两个类似的未解答的问题。直到我发布问题后我才注意到这些。

1. 通过Python脚本调用spider时，JSON在scrapy中不起作用？
2. 从 python 脚本调用 scrapy 不创建 JSON 输出文件

这是我从脚本运行 scrapy 的代码。它包括使用管道和 feed 导出器打印输出文件的设置。

from twisted.internet import reactor

from scrapy import log, signals
from scrapy.crawler import Crawler
from scrapy.xlib.pydispatch import dispatcher
import logging

from external_links.spiders.test import MySpider
from scrapy.utils.project import get_project_settings
settings = get_project_settings()

#manually set settings here
settings.set('ITEM_PIPELINES',{'external_links.pipelines.FilterPipeline':100,'external_links.pipelines.CsvWriterPipeline':200},priority='cmdline')
settings.set('DEPTH_LIMIT',1,priority='cmdline')
settings.set('LOG_FILE','Log.log',priority='cmdline')
settings.set('FEED_URI','output.csv',priority='cmdline')
settings.set('FEED_FORMAT', 'csv',priority='cmdline')
settings.set('FEED_EXPORTERS',{'csv':'external_links.exporter.CsvOptionRespectingItemExporter'},priority='cmdline')
settings.set('FEED_STORE_EMPTY',True,priority='cmdline')

def stop_reactor():
    reactor.stop()

dispatcher.connect(stop_reactor, signal=signals.spider_closed)
spider = MySpider()
crawler = Crawler(settings)
crawler.configure()
crawler.crawl(spider)
crawler.start() …

我正在学习Twisted,初学者的教程经常使用Factory和Protocol作为示例.工厂和协议接口似乎不支持发送消息.是否期望发送消息独立于协议接口实现？

class IProcessProtocol(Interface):
    def makeConnection(process):
    def childDataReceived(childFD, data):
    def childConnectionLost(childFD):
    def processExited(reason):
    def processEnded(reason):

我是python的新手,我正在编写一个简单的TCP服务器Server.py,我正在尝试使用这行代码导入反应堆 from twisted.internet import reactor 问题是当我运行代码时出现此错误

/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5     /Users/Nora/PycharmProjects/Server/Server.py
Traceback (most recent call last):
  File "/Users/Nora/PycharmProjects/Server/Server.py", line 2, in <module>
    from twisted.internet import reactor
ImportError: No module named 'twisted'

请注意,我使用的是OS X Yosemite,我已经安装了一个新的python版本3.5,我怎样才能回到已经扭曲的版本内置的系统？

我想ReconnectingClientFactory用 asyncio做一个。特别是处理客户端启动时服务器不可用的情况，在这种情况下ReconnectingClientFactory将继续尝试。这是asyncio.events.create_connection不做的事情。

具体来说：

该EchoClient例子就可以了。关键是如何建立连接。

factory = EchoClientFactory('ws://127.0.0.1:5678')
connectWS(factory)

在扭曲版本的情况下ReconnectingClientFactory。

对比

factory = EchoClientFactory(u"ws://127.0.0.1:5678")
factory.protocol = SecureServerClientProtocol

loop = asyncio.get_event_loop()

# coro = loop.create_connection(factory, 'ws_server', 5678)
coro = loop.create_connection(factory, '127.0.0.1', 5678)

loop.run_until_complete(asyncio.wait([
    alive(), coro
]))
loop.run_forever()
loop.close()

或与asycnio版本类似。

问题在于，在 asyncio 版本中，asyncio.events.create_connection如果服务器不可用，连接是通过它建立的。

我该如何调和两者？

非常感谢

我有这个简单的Twisted Client连接到Twisted服务器并查询索引.如果你看到fn.connectionMade()在class SpellClient,它query是硬编码的.这是为了测试目的.如何将此查询从外部传递给此类？

代码 -

from twisted.internet import reactor
from twisted.internet import protocol

# a client protocol
class SpellClient(protocol.Protocol):
    """Once connected, send a message, then print the result."""

    def connectionMade(self):
        query = 'abased'
        self.transport.write(query)

    def dataReceived(self, data):
        "As soon as any data is received, write it back."
        print "Server said:", data
        self.transport.loseConnection()

    def connectionLost(self, reason):
        print "connection lost"

class SpellFactory(protocol.ClientFactory):
    protocol = SpellClient

    def clientConnectionFailed(self, connector, reason):
        print "Connection failed - goodbye!"
        reactor.stop()

    def …

我想了解以下2个代码段的内容-

片段＃1

from twisted.internet import threads, defer, reactor

def proc1(a):
    while True:
        print "Proc----------1"

def proc2(a):
    while True:
        print "Proc----------2"

def proc3(a):
    while True:
        print "Proc----------3"

d1 = threads.deferToThread(proc1)
d2 = threads.deferToThread(proc2)
d3 = threads.deferToThread(proc3)

reactor.run()

我的理解是所有线程并行运行，并且输出是所有混合的proc的=> stdout

快照者＃2

from twisted.internet import threads, defer, reactor

def proc1(a):
    while True:
        print "Proc----------1"

def proc2(a):
    while True:
        print "Proc----------2"

def proc3(a):
    while True:
        print "Proc----------3"



d1 = defer.Deferred()
d2 = defer.Deferred()
d3 = defer.Deferred()

d1.addCallback(proc1)
d2.addCallback(proc2)
d3.addCallback(proc3)

d1.callback('a')
d2.callback('a')
d3.callback('a')

reactor.run()

对于此代码段-每个延迟的回调都一个接一个地触发，就输出而言，只有proc1 …