如何在python中使用urllib2加速获取页面？

Question

我有一个脚本,可以获取多个网页并解析信息.

(可以在http://bluedevilbooks.com/search/?DEPT=MATH&CLASS=103&SEC=01上看到一个例子)

我在它上面运行了cProfile,而且正如我所假设的那样,urlopen占用了大量的时间.有没有办法更快地获取页面？或者一种方法一次获取几个页面？我会做最简单的事情,因为我是python和web开发的新手.

提前致谢!:)

更新:我有一个调用的函数fetchURLs(),我用它来制作一个我需要的URL数组,如下所示urls = fetchURLS().这些URL是来自亚马逊和eBay API的所有XML文件(这让我很困惑,为什么加载需要这么长时间,也许我的虚拟主机很慢？)

我需要做的是加载每个URL,读取每个页面,并将该数据发送到脚本的另一部分,该部分将解析和显示数据.

请注意,在获取所有页面之前,我无法执行后一部分,这就是我的问题所在.

另外,我相信我的主机一次限制我25个进程,所以服务器上最简单的任何东西都会很好:)

---

这是时间:

Sun Aug 15 20:51:22 2010    prof

         211352 function calls (209292 primitive calls) in 22.254 CPU seconds

   Ordered by: internal time
   List reduced from 404 to 10 due to restriction <10>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       10   18.056    1.806   18.056    1.806 {_socket.getaddrinfo}
     4991    2.730    0.001    2.730    0.001 {method 'recv' of '_socket.socket' objects}
       10    0.490    0.049    0.490    0.049 {method 'connect' of '_socket.socket' objects}
     2415    0.079    0.000    0.079    0.000 {method 'translate' of 'unicode' objects}
       12    0.061    0.005    0.745    0.062 /usr/local/lib/python2.6/HTMLParser.py:132(goahead)
     3428    0.060    0.000    0.202    0.000 /usr/local/lib/python2.6/site-packages/BeautifulSoup.py:1306(endData)
     1698    0.055    0.000    0.068    0.000 /usr/local/lib/python2.6/site-packages/BeautifulSoup.py:1351(_smartPop)
     4125    0.053    0.000    0.056    0.000 /usr/local/lib/python2.6/site-packages/BeautifulSoup.py:118(setup)
     1698    0.042    0.000    0.358    0.000 /usr/local/lib/python2.6/HTMLParser.py:224(parse_starttag)
     1698    0.042    0.000    0.275    0.000 /usr/local/lib/python2.6/site-packages/BeautifulSoup.py:1397(unknown_starttag)

Answer 1

编辑:我正在扩大答案,包括一个更优秀的例子.我在这篇帖子中发现了很多关于线程与异步I/O的敌意和错误信息.因此,我还添加了更多的论据来驳斥某些无效的主张.我希望这将有助于人们为正确的工作选择合适的工具.

这是3天前的问题.

Python urllib2.open很慢,需要更好的方法来读取几个url - Stack Overflow Python urllib2.urlopen()很慢,需要更好的方法来读取几个url

我正在抛出代码以展示如何使用线程并行获取多个网页.

import time
import threading
import Queue

# utility - spawn a thread to execute target for each args
def run_parallel_in_threads(target, args_list):
    result = Queue.Queue()
    # wrapper to collect return value in a Queue
    def task_wrapper(*args):
        result.put(target(*args))
    threads = [threading.Thread(target=task_wrapper, args=args) for args in args_list]
    for t in threads:
        t.start()
    for t in threads:
        t.join()
    return result

def dummy_task(n):
    for i in xrange(n):
        time.sleep(0.1)
    return n

# below is the application code
urls = [
    ('http://www.google.com/',),
    ('http://www.lycos.com/',),
    ('http://www.bing.com/',),
    ('http://www.altavista.com/',),
    ('http://achewood.com/',),
]

def fetch(url):
    return urllib2.urlopen(url).read()

run_parallel_in_threads(fetch, urls)

如您所见,特定于应用程序的代码只有3行,如果您具有攻击性,可以将其折叠为1行.我认为任何人都不能证明他们声称这是复杂而不可维护的.

不幸的是,这里发布的大多数其他线程代码都有一些缺陷.他们中的许多人都进行主动轮询以等待代码完成.join()是一种更好的同步代码的方法.我认为此代码到目前为止已经改进了所有线程示例.

保持连接

如果所有URL指向同一服务器,WoLpH关于使用keep-alive连接的建议可能非常有用.

扭曲

Aaron Gallagher是twisted框架的粉丝,他建议线程的任何人都是敌对的.不幸的是,他的许多说法都是错误的信息.例如,他说"-1表示线程.这是IO绑定的;线程在这里没用." 这与证据相反,因为Nick T和我都证明了使用线程的速度增益.事实上,I/O绑定应用程序从使用Python的线程获得的收益最多(与CPU绑定应用程序中没有增益相比).Aaron对线程的误导性批评显示他对并行编程总体上感到困惑.

正确工作的正确工具

我很清楚使用线程,python,异步I/O等并行编程的问题.每个工具都有其优点和缺点.对于每种情况,都有适当的工具.我不反对扭曲(虽然我自己没有部署过).但我不相信我们可以说,在任何情况下,线程都是坏的,扭曲是好的.

例如,如果OP的要求是并行获取10,000个网站,则可以优先选择异步I/O. 线程不适用(除非使用无堆栈Python).

亚伦对线程的反对主要是概括.他没有意识到这是一项微不足道的并行化任务.每项任务都是独立的,不共享资源.所以他的大部分攻击都不适用.

鉴于我的代码没有外部依赖,我会称之为正确工作的正确工具.

性能

我想大多数人会同意这项任务的执行在很大程度上取决于网络代码和外部服务器,其中平台代码的性能应该可以忽略不计.然而,Aaron的基准测试表明,线程代码的速度提高了50%.我认为有必要回应这种明显的速度增益.

在Nick的代码中,有一个明显的缺陷导致效率低下.但是,您如何解释我的代码速度提升233ms？我认为即使扭曲的粉丝也不会跳到最后将这归因于扭曲的效率.毕竟,在系统代码之外存在大量变量,例如远程服务器的性能,网络,缓存以及urllib2和扭曲的Web客户端之间的差异实现等等.

为了确保Python的线程不会产生大量的低效率,我做了一个快速的基准来产生5个线程然后500个线程.我很自然地说产生5个线程的开销可以忽略不计,无法解释233ms的速度差异.

In [274]: %time run_parallel_in_threads(dummy_task, [(0,)]*5)
CPU times: user 0.00 s, sys: 0.00 s, total: 0.00 s
Wall time: 0.00 s
Out[275]: <Queue.Queue instance at 0x038B2878>

In [276]: %time run_parallel_in_threads(dummy_task, [(0,)]*500)
CPU times: user 0.16 s, sys: 0.00 s, total: 0.16 s
Wall time: 0.16 s

In [278]: %time run_parallel_in_threads(dummy_task, [(10,)]*500)
CPU times: user 1.13 s, sys: 0.00 s, total: 1.13 s
Wall time: 1.13 s       <<<<<<<< This means 0.13s of overhead

对我的并行获取进行的进一步测试表明,在17次运行中响应时间存在巨大差异.(不幸的是,我没有扭曲来验证Aaron的代码).

0.75 s
0.38 s
0.59 s
0.38 s
0.62 s
1.50 s
0.49 s
0.36 s
0.95 s
0.43 s
0.61 s
0.81 s
0.46 s
1.21 s
2.87 s
1.04 s
1.72 s

我的测试不支持Aaron的结论,即线程一直比异步I/O慢一个可测量的边缘.鉴于涉及的变量数量,我不得不说这不是衡量异步I/O和线程之间系统性能差异的有效测试.

Answer 2

使用扭曲!与使用线程相比,它使这种事情变得非常容易.

from twisted.internet import defer, reactor
from twisted.web.client import getPage
import time

def processPage(page, url):
    # do somewthing here.
    return url, len(page)

def printResults(result):
    for success, value in result:
        if success:
            print 'Success:', value
        else:
            print 'Failure:', value.getErrorMessage()

def printDelta(_, start):
    delta = time.time() - start
    print 'ran in %0.3fs' % (delta,)
    return delta

urls = [
    'http://www.google.com/',
    'http://www.lycos.com/',
    'http://www.bing.com/',
    'http://www.altavista.com/',
    'http://achewood.com/',
]

def fetchURLs():
    callbacks = []
    for url in urls:
        d = getPage(url)
        d.addCallback(processPage, url)
        callbacks.append(d)

    callbacks = defer.DeferredList(callbacks)
    callbacks.addCallback(printResults)
    return callbacks

@defer.inlineCallbacks
def main():
    times = []
    for x in xrange(5):
        d = fetchURLs()
        d.addCallback(printDelta, time.time())
        times.append((yield d))
    print 'avg time: %0.3fs' % (sum(times) / len(times),)

reactor.callWhenRunning(main)
reactor.run()

此代码的性能也优于任何其他已发布的解决方案(在我关闭一些使用大量带宽的内容后编辑):

Success: ('http://www.google.com/', 8135)
Success: ('http://www.lycos.com/', 29996)
Success: ('http://www.bing.com/', 28611)
Success: ('http://www.altavista.com/', 8378)
Success: ('http://achewood.com/', 15043)
ran in 0.518s
Success: ('http://www.google.com/', 8135)
Success: ('http://www.lycos.com/', 30349)
Success: ('http://www.bing.com/', 28611)
Success: ('http://www.altavista.com/', 8378)
Success: ('http://achewood.com/', 15043)
ran in 0.461s
Success: ('http://www.google.com/', 8135)
Success: ('http://www.lycos.com/', 30033)
Success: ('http://www.bing.com/', 28611)
Success: ('http://www.altavista.com/', 8378)
Success: ('http://achewood.com/', 15043)
ran in 0.435s
Success: ('http://www.google.com/', 8117)
Success: ('http://www.lycos.com/', 30349)
Success: ('http://www.bing.com/', 28611)
Success: ('http://www.altavista.com/', 8378)
Success: ('http://achewood.com/', 15043)
ran in 0.449s
Success: ('http://www.google.com/', 8135)
Success: ('http://www.lycos.com/', 30349)
Success: ('http://www.bing.com/', 28611)
Success: ('http://www.altavista.com/', 8378)
Success: ('http://achewood.com/', 15043)
ran in 0.547s
avg time: 0.482s

并使用Nick T的代码,操纵也可以给出平均值5并更好地显示输出:

Starting threaded reads:
...took 1.921520 seconds ([8117, 30070, 15043, 8386, 28611])
Starting threaded reads:
...took 1.779461 seconds ([8135, 15043, 8386, 30349, 28611])
Starting threaded reads:
...took 1.756968 seconds ([8135, 8386, 15043, 30349, 28611])
Starting threaded reads:
...took 1.762956 seconds ([8386, 8135, 15043, 29996, 28611])
Starting threaded reads:
...took 1.654377 seconds ([8117, 30349, 15043, 8386, 28611])
avg time: 1.775s

Starting sequential reads:
...took 1.389803 seconds ([8135, 30147, 28611, 8386, 15043])
Starting sequential reads:
...took 1.457451 seconds ([8135, 30051, 28611, 8386, 15043])
Starting sequential reads:
...took 1.432214 seconds ([8135, 29996, 28611, 8386, 15043])
Starting sequential reads:
...took 1.447866 seconds ([8117, 30028, 28611, 8386, 15043])
Starting sequential reads:
...took 1.468946 seconds ([8153, 30051, 28611, 8386, 15043])
avg time: 1.439s

并使用Wai Yip Tung的代码:

Fetched 8117 from http://www.google.com/
Fetched 28611 from http://www.bing.com/
Fetched 8386 from http://www.altavista.com/
Fetched 30051 from http://www.lycos.com/
Fetched 15043 from http://achewood.com/
done in 0.704s
Fetched 8117 from http://www.google.com/
Fetched 28611 from http://www.bing.com/
Fetched 8386 from http://www.altavista.com/
Fetched 30114 from http://www.lycos.com/
Fetched 15043 from http://achewood.com/
done in 0.845s
Fetched 8153 from http://www.google.com/
Fetched 28611 from http://www.bing.com/
Fetched 8386 from http://www.altavista.com/
Fetched 30070 from http://www.lycos.com/
Fetched 15043 from http://achewood.com/
done in 0.689s
Fetched 8117 from http://www.google.com/
Fetched 28611 from http://www.bing.com/
Fetched 8386 from http://www.altavista.com/
Fetched 30114 from http://www.lycos.com/
Fetched 15043 from http://achewood.com/
done in 0.647s
Fetched 8135 from http://www.google.com/
Fetched 28611 from http://www.bing.com/
Fetched 8386 from http://www.altavista.com/
Fetched 30349 from http://www.lycos.com/
Fetched 15043 from http://achewood.com/
done in 0.693s
avg time: 0.715s

我必须说,我确实喜欢顺序提取对我来说更好.

Answer 3

这是一个使用python的例子Threads.这里的其他线程示例每个url启动一个线程,如果它导致服务器处理的命中次数过多(例如,蜘蛛在同一主机上有很多url),这不是非常友好的行为

from threading import Thread
from urllib2 import urlopen
from time import time, sleep

WORKERS=1
urls = ['http://docs.python.org/library/threading.html',
        'http://docs.python.org/library/thread.html',
        'http://docs.python.org/library/multiprocessing.html',
        'http://docs.python.org/howto/urllib2.html']*10
results = []

class Worker(Thread):
    def run(self):
        while urls:
            url = urls.pop()
            results.append((url, urlopen(url).read()))

start = time()
threads = [Worker() for i in range(WORKERS)]
any(t.start() for t in threads)

while len(results)<40:
    sleep(0.1)
print time()-start

注意:此处给出的时间是40个网址,并且很大程度上取决于您的互联网连接速度和服务器延迟.在澳大利亚,我的ping是> 300ms

随着WORKERS=1花了86秒运行
有了WORKERS=4它花了23秒,运行
与WORKERS=10它花了10秒运行

所以下载10个线程的速度是单个线程的8.6倍.

这是使用队列的升级版本.至少有几个优点.
1.网址按照它们出现在列表中的顺序被请求
2.可用于q.join()检测请求何时全部完成
3.结果与网址列表保持相同的顺序

from threading import Thread
from urllib2 import urlopen
from time import time, sleep
from Queue import Queue

WORKERS=10
urls = ['http://docs.python.org/library/threading.html',
        'http://docs.python.org/library/thread.html',
        'http://docs.python.org/library/multiprocessing.html',
        'http://docs.python.org/howto/urllib2.html']*10
results = [None]*len(urls)

def worker():
    while True:
        i, url = q.get()
        # print "requesting ", i, url       # if you want to see what's going on
        results[i]=urlopen(url).read()
        q.task_done()

start = time()
q = Queue()
for i in range(WORKERS):
    t=Thread(target=worker)
    t.daemon = True
    t.start()

for i,url in enumerate(urls):
    q.put((i,url))
q.join()
print time()-start