Python:读取和拆分文件的快速方法是什么？

Question

我需要读取一个文件并将其拆分成行,并将这些行拆分为一半的制表符,以及删除所有语音标记.目前我有一个工作职能.但是,它很慢:

temp = []
fp = open(fName, "r")
for line in fp:
    temp.append(line.replace("\"","").rstrip("\n").split("\t"))
print temp

这会将文件拆分为列表列表.它实际上可能只是一个列表,因为只要订单被保留,以后很容易将它重新分配成对.

必须有更快的方法来做到这一点.谁能让我走上正轨？

谢谢!

[编辑]我正在使用的文件是巨大的,但我会添加类似的东西.(有没有办法在堆栈溢出上传文件？)

"CARMILLA"  "35"
"JONATHAN R"    "AA2"
"M" "3"
"EMMA"  "350"
"OLD"   "AA"

应该返回:

["CARMILLA", "35", "JONATHON R", "AA2", "M", "3", "EMMA", "350", "OLD", "AA"]

虽然我的代码将它作为2个字符串列表的列表返回,这也没关系.

对不起,我应该注意到print语句代表一个return语句 - 因为我把它从函数中取出来我把它改成了打印所以它在这里更有意义.

Answer 1

我认为列表理解会比调用.append每一行更快

from itertools import chain
with open('file.txt') as f:
    lines = chain.from_iterable([l.replace(r'"','').rstrip('\n').split('\t',1) for l in f])

编辑:所以它产生一个扁平的列表

>>> 
['CARMILLA', '35', 'JONATHAN R', 'AA2', 'M', '3', 'EMMA', '350', 'OLD', 'AA']

非展平版:

with open('file.txt') as f:
    lines = [l.replace(r'"','').rstrip('\n').split('\t',1) for l in f]

而一些时间,结果OP是最快的？

import timeit
print("chain, list",timeit.timeit(r"""
with open('file.txt') as f:
    lines = chain.from_iterable([l.replace(r'"','').rstrip('\n').split('\t',1) for l in f])""",setup="from itertools import chain",number=1000))
print("flat       ",timeit.timeit(r"""
with open('file.txt') as f:
    lines = [l.replace(r'"','').rstrip('\n').split('\t',1) for l in f]""",setup="from itertools import chain",number=1000))
print("op's       ",timeit.timeit(r"""temp = []
fp = open('file.txt', "r")
for line in fp:
    temp.append(line.replace("\"","").rstrip("\n").split("\t"))
""",number=1000))
print("jamlyks    ",timeit.timeit(r"""
with open('file.txt', 'rb') as f:
    r = csv.reader(f, delimiter=' ', skipinitialspace=True)
    list(chain.from_iterable(r))""",setup="from itertools import chain; import csv",number=1000))
print("lennart    ",timeit.timeit(r"""
    list(csv.reader(open('file.txt'), delimiter='\t', quotechar='"'))""",setup="from itertools import chain; import csv",number=1000))

产量

C:\Users\Henry\Desktop>k.py
('chain, list', 0.04725674146159321)
('my flat    ', 0.04629905135295972)
("op's       ", 0.04391255644624917)
('jamlyks    ', 0.048360870934994915)
('lennart    ', 0.04569112379085424)