相关疑难解决方法(0)

大型CSV文件(numpy)上的内存不足

我有一个3GB的CSV文件,我尝试用python读取,我需要明智的中间列.

from numpy import * 
def data():
    return genfromtxt('All.csv',delimiter=',')

data = data() # This is where it fails already.

med = zeros(len(data[0]))
data = data.T
for i in xrange(len(data)):
    m = median(data[i])
    med[i] = 1.0/float(m)
print med
Run Code Online (Sandbox Code Playgroud)

我得到的错误是这样的:

Python(1545) malloc: *** mmap(size=16777216) failed (error code=12)

*** error: can't allocate region

*** set a breakpoint in malloc_error_break to debug

Traceback (most recent call last):

  File "Normalize.py", line 40, in <module>

  data = data()

  File "Normalize.py", line 39, in data

  return genfromtxt('All.csv',delimiter=',') …
Run Code Online (Sandbox Code Playgroud)

python memory csv numpy scipy

33
推荐指数
1
解决办法
2万
查看次数

标签 统计

csv ×1

memory ×1

numpy ×1

python ×1

scipy ×1