Python线程:无法启动新线程

Question

我正在尝试运行此代码:

def VideoHandler(id):
    try:
        cursor = conn.cursor()
        print "Doing {0}".format(id)
        data = urllib2.urlopen("http://myblogfms2.fxp.co.il/video" + str(id) + "/").read()
        title = re.search("<span class=\"style5\"><strong>([\\s\\S]+?)</strong></span>", data).group(1)
        picture = re.search("#4F9EFF;\"><img src=\"(.+?)\" width=\"120\" height=\"90\"", data).group(1)
        link = re.search("flashvars=\"([\\s\\S]+?)\" width=\"612\"", data).group(1)
        id = id
        print "Done with {0}".format(id)
        cursor.execute("insert into videos (`title`, `picture`, `link`, `vid_id`) values('{0}', '{1}', '{2}', {3})".format(title, picture, link, id))
        print "Added {0} to the database".format(id)
    except:
        pass

x = 1
while True:
    if x != 945719:
        currentX = x
        thread.start_new_thread(VideoHandler, (currentX))
    else:
        break
    x += 1

它说"无法启动新线程"

Answer 1

错误的真正原因很可能是您创建了太多线程(超过100k !!!)并达到操作系统级别的限制.

除此之外,您的代码可以通过多种方式进行改进:

• 不要使用低级thread模块,请使用模块中的Thread类threading.
• 加入代码末尾的线程
• 将您创建的线程数限制为合理的:处理所有元素,创建少量线程,让每个线程处理整个数据的子集(这是我在下面提出的建议,但您也可以采用生产者 - 消费者工作线程从queue.Queue实例获取数据的模式)
• 和从未,曾经有一个except: pass在你的代码语句.或者如果你这样做,如果你的代码不起作用而你无法弄清楚原因,请不要在这里哭泣.:-)

这是一个提案:

from threading import Thread
import urllib2
import re

def VideoHandler(id_list):
    for id in id_list:
        try:
            cursor = conn.cursor()
            print "Doing {0}".format(id)
            data = urllib2.urlopen("http://myblogfms2.fxp.co.il/video" + str(id) + "/").read()
            title = re.search("<span class=\"style5\"><strong>([\\s\\S]+?)</strong></span>", data).group(1)
            picture = re.search("#4F9EFF;\"><img src=\"(.+?)\" width=\"120\" height=\"90\"", data).group(1)
            link = re.search("flashvars=\"([\\s\\S]+?)\" width=\"612\"", data).group(1)
            id = id
            print "Done with {0}".format(id)
            cursor.execute("insert into videos (`title`, `picture`, `link`, `vid_id`) values('{0}', '{1}', '{2}', {3})".format(title, picture, link, id))
            print "Added {0} to the database".format(id)
        except:
            import traceback
            traceback.print_exc()

conn = get_some_dbapi_connection()         
threads = []
nb_threads = 8
max_id = 945718
for i in range(nb_threads):
    id_range = range(i*max_id//nb_threads, (i+1)*max_id//nb_threads + 1)
    thread = Thread(target=VideoHandler, args=(id_range,))
    threads.append(thread)
    thread.start()

for thread in threads:
    thread.join() # wait for completion