使用简单的python生成器作为Tornado异步处理程序中的协同例程？

Question

我有一个python生成器函数,它产生大量的文本.我想get为tornado.web.RequestHandler子类编写一个方法,它将遍历生成器,在响应时将块写入响应.

因为这是Tornado,并且由于生成器可能需要一秒钟来处理,我认为使处理程序异步,使用此生成器作为协同例程并在每个块之后将控制权传递给IOLoop会很好.但是,我无法做出如何做到这一点的正面或反面.

这是我的示例(阻止)代码:

class TextHandler(web.RequestHandler):
    @web.asynchronous
    def get(self, n):
        generator = self.generate_text(100000)
        # Clearly, this will block. How to make it asynchronous?
        for text in generator:
            self.write(text)

    def generate_text(n):
        for x in xrange(n):
            if not x % 15:
                yield "FizzBuzz\n"
            elif not x % 5:
                yield "Buzz\n"
            elif not x % 3:
                yield "Fizz\n"
            else:
                yield "%s\n" % x

如何使此处理程序异步工作？

Answer 1

这是您所描述的基本版本.为避免阻塞,您可以通过回调函数将生成器传递给IOLoop.这里的诀窍是因为你没有使用实际IO的进程,因此没有os级进程/文件处理程序来添加到IOLoop via add_handler,你可以改为使用一个简单的add_callback调用并在回调函数中重复调用它来保持IOLoop回调队列中的函数,直到生成器完成.

import tornado.httpserver
import tornado.ioloop
import tornado.web

class TextHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    def get(self):
        self.generator = self.generate_text(1000)
        tornado.ioloop.IOLoop.instance().add_callback(self.loop)

    def loop(self):
        try:
            text = self.generator.next()
            self.write(text)
            tornado.ioloop.IOLoop.instance().add_callback(self.loop)
        except StopIteration:
            self.finish()

    def generate_text(self, n):
        for x in xrange(n):
            if not x % 15:
                yield "FizzBuzz\n"
            elif not x % 5:
                yield "Buzz\n"
            elif not x % 3:
                yield "Fizz\n"
            else:
                yield "%s\n" % x

application = tornado.web.Application([
    (r"/text/", TextHandler),
])

http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()

Answer 2

也可以将新龙卷风的gen接口用于异步进程:

import tornado.httpserver
import tornado.ioloop
import tornado.web
import tornado.gen

class TextHandler(tornado.web.RequestHandler):

    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):

        def cb(it, callback):
            try:
                value = it.next()
            except StopIteration:
                value = None
            callback(value)

        it = self.generate_text(1000)
        while True:
            response = yield tornado.gen.Task(cb, it)
            if response:
                self.write(response)
            else:
                break
        self.finish()

    def generate_text(self, n):
        for x in xrange(n):
            if not x % 15:
                yield "FizzBuzz\n"
            elif not x % 5:
                yield "Buzz\n"
            elif not x % 3:
                yield "Fizz\n"
            else:
                yield "%s\n" % x

application = tornado.web.Application([
    (r"/text/", TextHandler),
])

http_server = tornado.httpserver.HTTPServer(application)
http_server.listen(8888)
tornado.ioloop.IOLoop.instance().start()