小编dog*_*ogs的帖子

Pytorch 从张量文件中读取张量(从磁盘进行流训练)

我有一些非常大的输入张量,在构建它们时遇到了内存问题,所以我将它们一一读入文件中.pt。当我运行生成并保存文件的脚本时,文件变得越来越大,因此我假设张量正确保存。这是该代码:

with open(a_sync_save, "ab") as f:
     print("saved")
     torch.save(torch.unsqueeze(torch.cat(tensors, dim=0), dim=0), f)
Run Code Online (Sandbox Code Playgroud)

我想一次从文件中读取一定数量的张量,因为我不想再次遇到内存问题。当我尝试读取保存到文件中的每个张量时,我只能设法获取第一个张量。

with open(a_sync_save, "rb") as f:
    for tensor in torch.load(f):
        print(tensor.shape)
Run Code Online (Sandbox Code Playgroud)

这里的输出是第一个张量的形状,然后平静退出。

python-3.x torch pytorch

5
推荐指数
1
解决办法
1075
查看次数

如何使用 Sklearn.preprocessing 对包含列表的 pandas.DataFrame 列进行编码

我有一个 pandas df ,其中一些列是其中包含数据的列表,我想对列表中的标签进行编码。

我收到此错误: ValueError: Expected 2D array, got 1D array instead:

from sklearn.preprocessing import OneHotEncoder
mins = pd.read_csv('recipes.csv')

enc = OneHotEncoder(handle_unknown='ignore')

X = mins['Ingredients']

'''
[[lettuce, tomatoes, ginger, vodka, tomatoes]
[lettuce, tomatoes, flour, vodka, tomatoes]
...
[flour, tomatoes, vodka, vodka, mustard]]
'''

enc.fit(X)
Run Code Online (Sandbox Code Playgroud)

我希望得到一列包含正确编码信息的列表

[[lettuce, tomatoes, ginger, vodka, tomatoes]
[lettuce, tomatoes, flour, vodka, tomatoes]
...
[flour, tomatoes, vodka, vodka, mustard]

[[0, 1, 2, 3, 1]
[0, 1, 4, 3, 1]
...
[4, 1, 3, 3, 9]]
Run Code Online (Sandbox Code Playgroud)

python dataframe python-3.x scikit-learn sklearn-pandas

4
推荐指数
1
解决办法
2003
查看次数

在 tensorflow 2 中加载 ModelCheckpoint

In keras using tensorflow 1 I could ModelCheckpoint(filepath) and the saved file was a called filepath and then I could call model = load_model(filepath) to load the saved model.

Now the equivalent in tensorflow 2 the ModelCheckpoint creates a directory called filepath and when I follow the documentation here to load the saved model I have to create an empty model then call model.load_weights(filepath). Here is my callback and fit:


filepath = "filepath"
checkpoint = tf.keras.callbacks.ModelCheckpoint(filepath=filepath, mode='max', monitor='val_accuracy', verbose=2, …
Run Code Online (Sandbox Code Playgroud)

python-3.x keras tensorflow tf.keras tensorflow2.0

2
推荐指数
1
解决办法
1516
查看次数