通过附加数据构建 Numpy 数组（事先不知道完整大小）

Question

我有很多文件，每个文件都被读取为 shape 的矩阵(n, 1000)，其中 n 可能因文件而异。

我想将它们全部连接成一个大的 Numpy 数组。我目前这样做：

dataset = np.zeros((100, 1000))
for f in glob.glob('*.png'):
    x = read_as_numpyarray(f)    # custom function; x is a matrix of shape (n, 1000)
    dataset = np.vstack((dataset, x))

但它效率低下，因为我dataset通过将现有数组与读取的下一个文件堆叠起来多次重新定义。

如何使用 Numpy 以更好的方式做到这一点，避免整个数据集在内存中多次重写？

注意：最终的大 Numpy 数组可能需要 10 GB。

Answer 1

使用原生list的 numpy 数组，然后np.concatenate.

本机list将在需要时将大小乘以（〜1.125），因此不会发生太多的重新分配，而且，它只会保存指向分散（内存中不连续）的np.arrays实际数据的指针。

concatenate只需拨打一次即可解决您的问题。

伪代码

dataset = []
for f in glob.glob('*.png'):
    x = read_as_numpyarray(f)    # custom function; x is a matrix of shape (n, 1000)
    dataset.append(x)

dataset_np = np.concatenate(dataset)

注意vstack内部使用concatenate.

编辑以解决已编辑的问题：

假设数据总大小为 20 GB。连接时，系统仍必须保留 20 GB（对于每个单独的数组），并为新的连接数组分配 20 GB，因此需要 40 GB RAM（数据集的两倍）。如何在不需要双倍 RAM 的情况下做到这一点？（例如：如果我们只有 32 GB RAM，有没有解决方案？）

I would attack this problem first by doing the same as proposed in this current answer, in phases. dataset_np1 = np.concatenate(half1_of_data), dataset_np2 = np.concatenate(half2_of_data), will only need 150% RAM (not 200%). This can be extended recursively at the expense of speed until the limit at which this becomes the proposition in the question. I can only assume the likes of dask can handle this better, but haven't tested myself.

Just to clarify, after you have dataset_np1 you no longer need the list of all the sharded small arrays, and can free that. Only then you start loading the other half. Thus, you only ever need to hold an extra 50% of the data in memory.

Pseudocode:

def load_half(buffer: np.array, shard_path: str, shard_ind: int):
    half_dataset = []
    for f in glob.glob(f'{shard_path}/*.png'):
        x = read_as_numpyarray(f)    # custom function; x is a matrix of shape (n, 1000)
        half_dataset.append(x)

    half_dataset_np = np.concatenate(half_dataset) # see comment *
    buffer[:buffer.shape[0] // 2 * (shard_ind + 1), ...] = half_dataset_np

half1_path = r"half1"  # preprocess the shards to be found by glob or otherwise
half2_path = r"half2"
assert os.path.isdir(half1_path)
assert os.path.isdir(half2_path)

buffer = np.zeros(size_shape)
half1_np = load_half(half1_path, buffer, 0) # only 50% of data temporarily loaded, then freed [can be done manually if needed]
half2_np = load_half(half2_path, buffer, 1) # only 50% of data temporarily loaded, then freed

One could (easily, or not so easily) generalize this to quarters, eighths, or recursively any required fraction to reduce memory costs at the expense of speed, with the limit at infinity being the original proposition in the question.

• Important comment (see "see comment * in the code):
  One might notice half_dataset_np = np.concatenate(half_dataset) actually allocates 50% of the dataset, with the other 50% allocated in shards, apparently saving us nothing. That is correct, and I could not find a way to concat into a buffer. However, implementing this recursively as suggested (and not shown in pseudocode) will save memory, as a quarter will only use 2* 25% every time. This is just an implementation detail, but I hope the gist is clear.

On a different note, another approach would state "what if the dataset is 1000GB"? then no numpy array will do. This is why databases exist, and they can be queried quite efficiently using tools. But again, this is somewhat a research question, and depends heavily on your specific needs. As a very uninformed hunch, I would check out dask.

Such libraries will obviously tackle problems like this one as a subset of what they do, and I recommend not implementing these things yourself, as the total time you will spend will much outweigh the time choosing and learning a library.

On another different note, I wonder if this really has to be such a huge array, and maybe a slightly different design or formulation of the problem could alleviate us from this technical issue altogether.