文件太大而无法读取

Question

我有一个大小为3,8GB的文件“ uniprot.tab”。

我正在尝试根据此文件绘制直方图，但由于它太大而无法完成计算。

之前，我已经用一个小文件“ mock.tab”测试了我的代码，它可以正常工作。

编辑：例如“ mock.dat”的某些行：

Entry   Status  Cross-reference (PDB)
A1WYA9  reviewed    
Q6LLK1  reviewed    
Q1ACM9  reviewed    
P10994  reviewed    1OY8;1OY9;1OY9;1OY9;
Q0HV56  reviewed    
Q2NQJ2  reviewed    
B7HCE7  reviewed    
P0A959  reviewed    4CVQ;
B7HLI3  reviewed    
P31224  reviewed    1IWG;1OY6;1OY8;1OY9;4CVQ;

在这里，您可以看到小文件上使用的代码：

import matplotlib.pyplot as plt

occurrences = []
with open('/home/martina/Documents/webstormProj/unpAnalysis/mock.tab', 'r') as f:
    next(f) #do not read the heading
    for line in f:
        col_third = line.split('\t')[2] #take third column
        occ = col_third.count(';') # count how many times it finds ; in each line
        occurrences.append(occ)

x_min = min(occurrences)
x_max = max(occurrences)


x = [] # x-axis
x = list(range(x_min, x_max + 1))

y = [] # y-axis
for i in x:
    y.append(occurrences.count(i))

plt.bar(x,y,align='center') # draw the plot
plt.xlabel('Bins')
plt.ylabel('Frequency')
plt.show()

如何使用大文件绘制此图？

Answer 1

与其构建所有值的列表，然后为每个值计数出现次数，不如直接在迭代时构建直方图，将更快。您可以collections.Counter为此使用：

from collections import Counter

histogram = Counter()
with open(my_file, 'r') as f:
    next(f)
    for line in file:
        # split line, etc. 
        histogram[occ] += 1

# now histogram is a dictionary containing each "occurrence" value and the count of how many times it was seen.

x_axis = list(range(min(histogram), max(histogram)+1))
y_axis = [histogram[x] for x in x_axis]