来自csv文件的numpy数组为千层面

Question

我开始学习如何使用带有千层面的theano,并从mnist示例开始.现在,我想尝试一下我自己的例子:我有一个train.csv文件,其中每一行以0或1开头,代表正确的答案,然后是773 0和1代表输入.我不明白如何将此文件转换为load_database()函数中所需的numpy数组.这是mnist数据库的原始函数的一部分:

...

with gzip.open(filename, 'rb') as f:
    data = pickle_load(f, encoding='latin-1')

# The MNIST dataset we have here consists of six numpy arrays:
# Inputs and targets for the training set, validation set and test set.
X_train, y_train = data[0]
X_val, y_val = data[1]
X_test, y_test = data[2]

...

# We just return all the arrays in order, as expected in main().
# (It doesn't matter how we do this as long as we can read them again.)
return X_train, y_train, X_val, y_val, X_test, y_test

我需要从我的csv文件中获取X_train(输入)和y_train(每行的开头).

谢谢!

Answer 1

您可以使用numpy.genfromtxt()或 ，numpy.loadtxt()如下所示：

from sklearn.cross_validation import KFold

Xy = numpy.genfromtxt('yourfile.csv', delimiter=",")

# the next section provides the required
# training-validation set splitting but 
# you can do it manually too, if you want

skf = KFold(len(Xy))

for train_index, valid_index in skf:
    ind_train, ind_valid = train_index, valid_index
    break

Xy_train, Xy_valid = Xy[ind_train], Xy[ind_valid]

X_train = Xy_train[:, 1:]
y_train = Xy_train[:, 0]

X_valid = Xy_valid[:, 1:]
y_valid = Xy_valid[:, 0]


...

# you can simply ignore the test sets in your case
return X_train, y_train, X_val, y_val #, X_test, y_test

在代码片段中，我们忽略了传递test集合。

现在您可以将数据集导入主模块或脚本或其他任何内容，但请注意从中删除所有测试部分。

或者，您也可以简单地将有效集作为testset 传递：

# you can simply pass the valid sets as `test` set
return X_train, y_train, X_val, y_val, X_val, y_val

在后一种情况下，我们不必关心涉及例外集的主要模块部分test，但作为分数（如果有），您将得到两次validation scores，即 as test scores。

注意：我不知道哪个 mnist 示例是这样的，但可能在您按照上述方式准备好数据后，您还必须在训练器模块中进行进一步修改以适合您的数据。例如：数据的输入形状，输出形状即类的数量，例如在您的情况下，前者是773，后者是2。