Python中巨大矩阵的矩阵运算

Question

有人知道如何在python中处理巨大矩阵吗？我必须处理形状为（10 ^ 6,10 ^ 6）的邻接矩阵，并执行包括加法，缩放和点积的操作。使用numpy数组，我在ram上遇到了问题。

Answer 1

这样的事情怎么样...

import numpy as np

# Create large arrays x and y.
# Note they are 1e4 not 1e6 b/c of memory issues creating random numpy matrices (CookieOfFortune) 
# However, the same principles apply to larger arrays
x = np.random.randn(10000, 10000)
y = np.random.randn(10000, 10000)

# Create memory maps for x and y arrays
xmap = np.memmap('xfile.dat', dtype='float32', mode='w+', shape=x.shape)
ymap = np.memmap('yfile.dat', dtype='float32', mode='w+', shape=y.shape)

# Fill memory maps with data
xmap[:] = x[:]
ymap[:] = y[:]

# Create memory map for out of core dot product result
prodmap = np.memmap('prodfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core dot product and write data
prodmap[:] = np.memmap.dot(xmap, ymap)

# Create memory map for out of core addition result
addmap = np.memmap('addfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core addition and write data
addmap[:] = xmap + ymap

# Create memory map for out of core scaling result
scalemap = np.memmap('scalefile.dat', dtype='float32', mode='w+', shape=x.shape)

# Define scaling constant
scale = 1.3

# Do out of core  scaling and write data
scalemap[:] = scale * xmap

此代码将创建文件xfile.dat，yfile.dat等，这些文件包含二进制格式的数组。要稍后访问它们，您只需要做np.memmap(filename)。其他参数np.memmap是可选的，但推荐使用（诸如dtype，shape等的参数）。