小编swo*_*ers的帖子

我正在 Pytorch 中训练图像分类模型，并使用其默认数据加载器来加载我的训练数据。我有一个非常大的训练数据集，因此每个类通常有几千个样本图像。过去我已经用大约 20 万张图像训练过模型，没有出现任何问题。然而我发现，当图像总数超过一百万张时，Pytorch 数据加载器就会卡住。

我相信当我打电话时代码挂起datasets.ImageFolder(...)。当我按下 Ctrl-C 时，输出始终如下：

Traceback (most recent call last):                                                                                                 \xe2\x94\x82\n  File "main.py", line 412, in <module>                                                                                            \xe2\x94\x82\n    main()                                                                                                                         \xe2\x94\x82\n  File "main.py", line 122, in main                                                                                                \xe2\x94\x82\n    run_training(args.group, args.num_classes)                                                                                     \xe2\x94\x82\n  File "main.py", line 203, in run_training                                                                                        \xe2\x94\x82\n    train_loader = create_dataloader(traindir, tfm.train_trans, shuffle=True)                                                      \xe2\x94\x82\n  File "main.py", line 236, in create_dataloader                                                                                   \xe2\x94\x82\n    dataset = datasets.ImageFolder(directory, trans)                                                                               \xe2\x94\x82\n  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 209, in __init__     \xe2\x94\x82\n    is_valid_file=is_valid_file)                                                                                                   \xe2\x94\x82\n  File "/home/username/.local/lib/python3.5/site-packages/torchvision/datasets/folder.py", line 94, in __init__ …

我在一台机器上有64个内核,运行排序总计1GB的数据.它们每个排序156,250项,不应共享任何数据结构(即总共有64个单独的数组被排序).但是,我运行的核心越多,每个核心在其自己的排序任务中的速度就越慢.

时间测量正在这样做:

void sort_ranges(std::vector<std::vector<std::vector<int> > > & range_partitions, int num_workers, std::string filename, std::string outfile)
{
  #pragma omp parallel default(none) shared(range_partitions, outfile, num_workers)
  {
    int i = omp_get_thread_num();
    std::vector<int> data_vec; //Data copied into separate data structure for each thread
    for(int x = 0; x < num_workers; x ++) {
      data_vec.reserve(data_vec.size() + (range_partitions[x][i]).size());
      data_vec.insert(data_vec.end(), range_partitions[x][i].begin(), range_partitions[x][i].end());
    }
    int n = data_vec.size();
    int * data = &data_vec[0];
    double start = omp_get_wtime();
    std::sort(data, data + n); //Measure sort function call
    double sort_done = omp_get_wtime() …