任何线程完成任务时终止多个线程

Question

我是python和线程的新手.我编写了python代码,它充当网络爬虫,并在网站上搜索特定的关键字.我的问题是,如何使用线程同时运行我的类的三个不同实例.当其中一个实例找到关键字时,所有三个实例都必须关闭并停止对Web进行爬网.这是一些代码.

class Crawler:
      def __init__(self):
            # the actual code for finding the keyword 

 def main():  
        Crawl = Crawler()

 if __name__ == "__main__":
        main()

我如何使用线程让Crawler同时进行三次不同的抓取？

Answer 1

似乎没有(简单)方法在Python中终止线程.

以下是并行运行多个HTTP请求的简单示例:

import threading

def crawl():
    import urllib2
    data = urllib2.urlopen("http://www.google.com/").read()

    print "Read google.com"

threads = []

for n in range(10):
    thread = threading.Thread(target=crawl)
    thread.start()

    threads.append(thread)

# to wait until all three functions are finished

print "Waiting..."

for thread in threads:
    thread.join()

print "Complete."

通过额外的开销,您可以使用更强大的多进程方法,并允许您终止类似线程的进程.

我已经扩展了这个例子来使用它.我希望这对你有所帮助:

import multiprocessing

def crawl(result_queue):
    import urllib2
    data = urllib2.urlopen("http://news.ycombinator.com/").read()

    print "Requested..."

    if "result found (for example)":
        result_queue.put("result!")

    print "Read site."

processs = []
result_queue = multiprocessing.Queue()

for n in range(4): # start 4 processes crawling for the result
    process = multiprocessing.Process(target=crawl, args=[result_queue])
    process.start()
    processs.append(process)

print "Waiting for result..."

result = result_queue.get() # waits until any of the proccess have `.put()` a result

for process in processs: # then kill them all off
    process.terminate()

print "Got result:", result

Answer 2

启动一个线程很简单:

thread = threading.Thread(function_to_call_inside_thread)
thread.start()

创建一个事件对象,以便在完成后通知:

event = threading.Event()
event.wait() # call this in the main thread to wait for the event
event.set() # call this in a thread when you are ready to stop

事件触发后,您需要向抓取工具添加stop()方法.

for crawler in crawlers:
    crawler.stop()

然后在线程上调用join

thread.join() # waits for the thread to finish

如果您进行任何此类编程,您将需要查看eventlet模块.它允许您编写"线程"代码,而没有线程的许多缺点.

Answer 3

首先,如果你是python的新手,我不建议面对线程.习惯语言,然后解决多线程问题.

话虽如此,如果你的目标是并行化(你说"同时运行"),你应该知道在python中(或者至少在默认实现中,CPython)多个线程不会真正并行运行,即使可提供多个处理器内核.阅读GIL(全球口译员锁)以获取更多信息.

最后,如果您还想继续,请查看线程模块的Python文档.我会说Python的文档和引用一样好,有大量的例子和解释.