将张量分成训练和测试集

Question

假设我使用的是在文本文件中读取的TextLineReader.有没有办法把它分成火车和测试集Tensorflow？就像是:

def read_my_file_format(filename_queue):
  reader = tf.TextLineReader()
  key, record_string = reader.read(filename_queue)
  raw_features, label = tf.decode_csv(record_string)
  features = some_processing(raw_features)
  features_train, labels_train, features_test, labels_test = tf.train_split(features,
                                                                            labels,
                                                                            frac=.1)
  return features_train, labels_train, features_test, labels_test

Answer 1

正如elham提到的,你可以使用scikit-learn轻松完成这项工作.scikit-learn是一个用于机器学习的开源库.有大量的数据准备工具,包括model_selection模块,用于处理比较,验证和选择参数.

该model_selection.train_test_split()方法专门用于随机和按百分比将数据拆分为训练集和测试集.

X_train, X_test, y_train, y_test = train_test_split(features,
                                                    labels,
                                                    test_size=0.33,
                                                    random_state=42)

test_size是为测试保留的百分比,random_state是随机抽样的种子.

我通常使用它来提供训练和验证数据集,并分别保存真实的测试数据.您也可以运行train_test_split两次来执行此操作.即将数据分成(训练+验证)和测试,然后将训练+验证分成两个单独的张量.

Answer 2

像下面这样的东西应该工作: tf.split_v(tf.random_shuffle(...

Answer 3

import sklearn.model_selection as sk

X_train, X_test, y_train, y_test = 
sk.train_test_split(features,labels,test_size=0.33, random_state = 42)