用Python预处理4亿条推文 - 速度更快

Question

我有4亿条推文(实际上我认为它几乎像450但不记得),形式如下:

T    "timestamp"
U    "username"
W    "actual tweet"

我想最初以"username\t tweet"的形式将它们写入文件,然后加载到DB中.问题是在加载到数据库之前,我需要做一些事情:1.预处理推文以删除RT @ [名称]和网址2.从http://twitter.com/中取出用户名用户名".

我正在使用python,这是代码.请让我知道如何更快地:)

'''The aim is  to take all the tweets of a user and store them in a table.  Do this for all the users and then lets see what we can do with it 
   What you wanna do is that you want to get enough information about a user so that you can profile them better. So , lets get started 
'''
def regexSub(line):
    line = re.sub(regRT,'',line)
    line = re.sub(regAt,'',line)
    line = line.lstrip(' ')
    line = re.sub(regHttp,'',line)
    return line
def userName(line):
    return line.split('http://twitter.com/')[1]


import sys,os,itertools,re
data = open(sys.argv[1],'r')
processed = open(sys.argv[2],'w')
global regRT 
regRT = 'RT'
global regHttp 
regHttp = re.compile('(http://)[a-zA-Z0-9]*.[a-zA-Z0-9/]*(.[a-zA-Z0-9]*)?')
global regAt 
regAt = re.compile('@([a-zA-Z0-9]*[*_/&%#@$]*)*[a-zA-Z0-9]*')

for line1,line2,line3 in itertools.izip_longest(*[data]*3):
    line1 = line1.split('\t')[1]
    line2 = line2.split('\t')[1]
    line3 = line3.split('\t')[1]

    #print 'line1',line1
    #print 'line2=',line2
    #print 'line3=',line3
    #print 'line3 before preprocessing',line3
    try:
        tweet=regexSub(line3)
        user = userName(line2)
    except:
        print 'Line2 is ',line2
        print 'Line3 is',line3

    #print 'line3 after processig',line3
    processed.write(user.strip("\n")+"\t"+tweet)

我按以下方式运行代码:

python -m cProfile -o profile_dump TwitterScripts/Preprocessing.py DATA/Twitter/t082.txt DATA/Twitter/preprocessed083.txt

这是我得到的输出:(警告:它相当大,我没有过滤掉小值,思考,它们也可能有一些意义)

Sat Jan  7 03:28:51 2012    profile_dump

         3040835560 function calls (3040835523 primitive calls) in 2500.613 CPU seconds

   Ordered by: call count

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
528840744  166.402    0.000  166.402    0.000 {method 'split' of 'str' objects}
396630560   81.300    0.000   81.300    0.000 {method 'get' of 'dict' objects}
396630560  326.349    0.000  439.737    0.000 /usr/lib64/python2.7/re.py:229(_compile)
396630558  255.662    0.000 1297.705    0.000 /usr/lib64/python2.7/re.py:144(sub)
396630558  602.307    0.000  602.307    0.000 {built-in method sub}
264420442   32.087    0.000   32.087    0.000 {isinstance}
132210186   34.700    0.000   34.700    0.000 {method 'lstrip' of 'str' objects}
132210186   27.296    0.000   27.296    0.000 {method 'strip' of 'str' objects}
132210186  181.287    0.000 1513.691    0.000 TwitterScripts/Preprocessing.py:4(regexSub)
132210186   79.950    0.000   79.950    0.000 {method 'write' of 'file' objects}
132210186   55.900    0.000  113.960    0.000 TwitterScripts/Preprocessing.py:10(userName)
  313/304    0.000    0.000    0.000    0.000 {len}

删除那些非常低的(如1,3等)

请告诉我可以做出哪些其他更改.谢谢 !

Answer 1

这就是多处理的目的.

您有一个可以分解为大量小步骤的管道.每个步骤都是Process从管道获取项目,进行小的转换并将中间结果放入下一个管道的步骤.

你将有一个Process读取原始文件一次三行,并将三行放入管道.就这样.

你将从Process管道获得一个(T,U,W)三元组,清理用户线,并将其放入下一个管道.

等等

不要构建太多的步骤来开始.读 - 转换 - 写是一个很好的开始,以确保你理解multiprocessing模块.之后,这是一项实证研究,以找出最佳的处理步骤组合.

当你解决这个问题时,它会产生许多通信顺序进程,这些进程将消耗你所有的CPU资源,但会相对快速地处理文件.

通常,同时工作的更多进程更快.由于操作系统开销和内存限制,您最终会达到限制.