大数组的RAM错误

Question

我需要随机获取一行的数字,并将每行放在其他数组中,然后获取一个col的数字.

我有一个超过400M的大文件.在该文件中,有13496*13496个数字,表示13496行和13496个列.我想把它们读成数组.这是我的代码:

_L1 = [[0 for col in range(13496)] for row in range(13496)]
_L1file = open('distanceCMD.function.txt')
while (i<13496):
    print "i="+str(i)
    _strlf = _L1file.readline()
    _strlf = _strlf.split('\t')
    _strlf = _strlf[:-1]
    _L1[i] = _strlf
    i += 1
_L1file.close()

这是我的错误信息:

MemoryError:
File "D:\research\space-function\ART3.py", line 30, in <module>
  _strlf = _strlf.split('\t')

Answer 1

你可能想以另一种方式处理你的问题.逐行处理文件.我认为不需要将整个大文件存储到数组中.否则,您可能想告诉我们您实际上要做什么.

for line in open("400MB_file"):
     # do something with line.

要么

f=open("file")
for linenum,line in enumerate(f):
    if linenum+1 in [2,3,10]:
         print "there are ", len(line.split())," columns" #assuming you want to split on spaces
         print "100th column value is: ", line.split()[99]
    if linenum+1>10:
         break # break if you want to stop after the 10th line
f.close()