Pytorch 数据加载器：错误的文件描述符和 EOF > 0

Question

问题描述

\n

在使用由自定义数据集制作的 Pytorch 数据加载器进行神经网络训练期间，我遇到了奇怪的行为。数据加载器设置为workers=4、pin_memory=False。

\n

大多数时候，训练都没有问题地完成。\n有时，训练会随机停止，并出现以下错误：

\n

\n
1. OSError: [Errno 9] 错误的文件描述符
\n
2. EOF错误
\n

\n

看起来错误发生在创建套接字以访问数据加载器元素期间。\n当我将工作线程数设置为 0 时，该错误消失，但我需要通过多处理来加速我的训练。\n错误的根源可能是什么？谢谢 ！

\n

python 和库的版本

\n

Python 3.9.12，Pyorch 1.11.0+cu102
\n编辑：该错误仅发生在集群上

\n

错误文件的输出

\n

Traceback (most recent call last):\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 145, in _serve\nEpoch 17:  52%|\xe2\x96\x88\xe2\x96\x88\xe2\x96\x88\xe2\x96\x88\xe2\x96\x88\xe2\x96\x8f    | 253/486 [01:00<00:55,  4.18it/s, loss=1.73]\n\nTraceback (most recent call last):\n  File "/my_directory/bench/run_experiments.py", line 251, in <module>\n    send(conn, destination_pid)\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 50, in send\n    reduction.send_handle(conn, new_fd, pid)\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 183, in send_handle\n    with socket.fromfd(conn.fileno(), socket.AF_UNIX, socket.SOCK_STREAM) as s:\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 545, in fromfd\n    return socket(family, type, proto, nfd)\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/socket.py", line 232, in __init__\n    _socket.socket.__init__(self, family, type, proto, fileno)\nOSError: [Errno 9] Bad file descriptor\n\n    main(args)\n  File "/my_directory/bench/run_experiments.py", line 183, in main\n    run_experiments(args, save_path)\n  File "/my_directory/bench/run_experiments.py", line 70, in run_experiments\n    ) = run_algorithm(algorithm_params[j], mp[j], ss, dataset)\n  File "/my_directorybench/algorithms.py", line 38, in run_algorithm\n    data = es(mp,search_space,  dataset, **ps)\n  File "/my_directorybench/algorithms.py", line 151, in es\n   data = ss.generate_random_dataset(mp,\n  File "/my_directorybench/architectures.py", line 241, in generate_random_dataset\n    arch_dict = self.query_arch(\n  File "/my_directory/bench/architectures.py", line 71, in query_arch\n    train_losses, val_losses, model = meta_net.get_val_loss(\n  File "/my_directory/bench/meta_neural_net.py", line 50, in get_val_loss\n    return self.training(\n  File "/my_directorybench/meta_neural_net.py", line 155, in training\n    train_loss = self.train_step(model, device, train_loader, epoch)\n  File "/my_directory/bench/meta_neural_net.py", line 179, in train_step\n    for batch_idx, mini_batch in enumerate(pbar):\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__\n    for obj in iterable:\n  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__\n    data = self._next_data()\n  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1207, in _next_data\n    idx, data = self._get_data()\n  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1173, in _get_data\n    success, data = self._try_get_data()\n  File "/my_directory/.local/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1011, in _try_get_data\n    data = self._data_queue.get(timeout=timeout)\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/queues.py", line 122, in get\n    return _ForkingPickler.loads(res)\n  File "/my_directory/.local/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 295, in rebuild_storage_fd\n    fd = df.detach()\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/resource_sharer.py", line 58, in detach\n    return reduction.recv_handle(conn)\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 189, in recv_handle\n    return recvfds(s, 1)[0]\n  File "/my_directory/.conda/envs/geoseg/lib/python3.9/multiprocessing/reduction.py", line 159, in recvfds\n    raise EOFError\nEOFError\n

\n

编辑：访问数据的方式

\n

    from PIL import Image\n    from torch.utils.data import DataLoader\n    \n    # extract of code of dataset\n    \n        class Dataset():\n           def __init__(self,image_files,mask_files):\n              self.image_files = image_files\n              self.mask_files = mask_files\n    \n           def __getitem__(self, idx):\n              img = Image.open(self.image_files[idx]).convert('RGB')\n              mask=Image.open(self.mask_files[idx]).convert('L')\n              return img, mask\n    \n    # extract of code of trainloader\n      \n        train_loader = DataLoader(\n                        dataset=train_dataset,\n                        batch_size=4,\n                        num_workers=4,\n                        pin_memory=False,\n                        shuffle=True,\n                        drop_last=True,\n                        persistent_workers=False,\n                    )\n

\n

Answer 1

我终于找到了解决方案。将此配置添加到数据集脚本中可以正常工作：

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')

默认情况下，共享策略设置为'file_descriptor'。

我尝试过一些解释的解决方案：

• 这个问题（增加共享内存，增加打开文件描述符的最大数量，每个时期结束时的 torch.cuda.empty_cache() ，...）
• 另一个问题，结果解决了这个问题

正如 @AlexMeredith 所建议的，该错误可能与某些集群使用的分布式文件系统（Lustre）有关。该错误也可能来自分布式共享内存。