Pytorch - 在使用 Dataloader 之前连接数据集

Question

我正在尝试加载两个数据集并将它们用于训练。

\n\n

软件包版本：python 3.7；\npytorch 1.3.1

\n\n

可以单独创建 data_loader 并按顺序对其进行训练：

\n\n

from torch.utils.data import DataLoader, ConcatDataset\n\n\ntrain_loader_modelnet = DataLoader(ModelNet(args.modelnet_root, categories=args.modelnet_categories,split=\'train\', transform=transform_modelnet, device=args.device),batch_size=args.batch_size, shuffle=True)\n\ntrain_loader_mydata = DataLoader(MyDataset(args.customdata_root, categories=args.mydata_categories, split=\'train\', device=args.device),batch_size=args.batch_size, shuffle=True)\n\nfor e in range(args.epochs):\n    for idx, batch in enumerate(tqdm(train_loader_modelnet)):\n        # training on dataset1\n    for idx, batch in enumerate(tqdm(train_loader_custom)):\n        # training on dataset2\n\n

\n\n

注意：MyDataset 是一个已def __len__(self): def __getitem__(self, index):实现的自定义数据集类。由于上述配置有效，看来这是实现正常的。

\n\n

但我理想情况下希望将它们组合成一个数据加载器对象。我按照 pytorch 文档尝试了这一点：

\n\n

train_modelnet = ModelNet(args.modelnet_root, categories=args.modelnet_categories,\n                          split=\'train\', transform=transform_modelnet, device=args.device)\ntrain_mydata = CloudDataset(args.customdata_root, categories=args.mydata_categories,\n                             split=\'train\', device=args.device)\ntrain_loader = torch.utils.data.ConcatDataset(train_modelnet, train_customdata)\n\nfor e in range(args.epochs):\n    for idx, batch in enumerate(tqdm(train_loader)):\n        # training on combined\n\n

\n\n

然而，在随机批次中，我得到以下“预期张量作为参数 0 中的元素 X，但得到了元组”类型的错误。任何帮助将非常感激！

\n\n

>   40%|\xe2\x96\x88\xe2\x96\x88\xe2\x96\x88\xe2\x96\x88      | 53/131 [01:03<02:00,  1.55s/it]\n>  Traceback (mostrecent call last):   File\n> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/pydevd.py",\n> line 1434, in _exec\n>     pydev_imports.execfile(file, globals, locals)  # execute the script   File\n> "/home/chris/Programs/pycharm-anaconda-2019.3.4/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile\n>     exec(compile(contents+"\\n", file, \'exec\'), glob, loc)   File "/home/chris/Documents/4yp/Data/my_kaolin/Classification/pointcloud_classification_combinedset.py",\n> line 83, in <module>\n>     for idx, batch in enumerate(tqdm(train_loader)):   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/tqdm/std.py",\n> line 1107, in __iter__\n>     for obj in iterable:   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/dataloader.py",\n> line 346, in __next__\n>     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration   File\n> "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py",\n> line 47, in fetch\n>     return self.collate_fn(data)   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",\n> line 79, in default_collate\n>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",\n> line 79, in <listcomp>\n>     return [default_collate(samples) for samples in transposed]   File "/home/chris/anaconda3/envs/4YP/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py",\n> line 55, in default_collate\n>     return torch.stack(batch, 0, out=out) TypeError: expected Tensor as element 3 in argument 0, but got tuple  \n

\n\n

\n

\n

Answer 1

如果我的问题正确，那么您的训练集和开发集（及其相应的加载器）如下：

train_set = CustomDataset(...)
train_loader = DataLoader(dataset=train_set, ...)
dev_set = CustomDataset(...)
dev_loader = DataLoader(dataset=dev_set, ...)

您想将它们连接起来以便使用 train+dev 作为训练数据，对吧？如果是这样，您只需调用：

train_dev_sets = torch.utils.data.ConcatDataset([train_set, dev_set])
train_dev_loader = DataLoader(dataset=train_dev_sets, ...)

这train_dev_loader是包含来自两组的数据的加载器。

现在，请确保您的数据具有相同的形状和相同的类型，即相同数量的特征，或相同的类别/数字等。

Answer 2

我猜这两个数据集有时会返回不同的类型。当数据是张量时，torch将它们堆叠起来，并且它们最好是相同的形状。如果它们是字符串之类的东西，火炬将从它们中创建一个元组。所以这听起来像是你的一个数据集有时返回的东西不是张量。我会在数据集的输出上放置一些断言，以检查它是否正在执行您想要的操作，或者深入研究pdb.