如何随机将数据拆分为trainset和testset？

Question

我有一个大型数据集,并希望将其分为培训(50%)和测试集(50%).

假设我有100个示例存储输入文件,每行包含一个示例.我需要选择50行作为训练集和50行测试集.

我的想法是首先生成一个长度为100的随机列表(值范围从1到100),然后使用前50个元素作为50个训练样例的行号.与测试集相同.

这可以在Matlab中轻松实现

fid=fopen(datafile);
C = textscan(fid, '%s','delimiter', '\n');
plist=randperm(100);
for i=1:50
    trainstring = C{plist(i)};
    fprintf(train_file,trainstring);
end
for i=51:100
    teststring = C{plist(i)};
    fprintf(test_file,teststring);
end

但是我怎么能在Python中完成这个功能呢？我是Python的新手,不知道我是否可以将整个文件读入数组,并选择某些行.

Answer 1

这可以使用列表在Python中类似地完成(注意整个列表是在适当的位置洗牌).

import random

with open("datafile.txt", "rb") as f:
    data = f.read().split('\n')

random.shuffle(data)

train_data = data[:50]
test_data = data[50:]

Answer 2

from sklearn.model_selection import train_test_split
import numpy

with open("datafile.txt", "rb") as f:
   data = f.read().split('\n')
   data = numpy.array(data)  #convert array to numpy type array

   x_train ,x_test = train_test_split(data,test_size=0.5)       #test_size=0.5(whole_data)

Answer 3

你也可以使用numpy.当您的数据存储在numpy.ndarray中时:

import numpy as np
from random import sample
l = 100 #length of data 
f = 50  #number of elements you need
indices = sample(range(l),f)

train_data = data[indices]
test_data = np.delete(data,indices)

Answer 4

你可以试试这个方法

import pandas
import sklearn
csv = pandas.read_csv('data.csv')
train, test = sklearn.cross_validation.train_test_split(csv, train_size = 0.5)

更新：train_test_split已转移到model_selection当前的方式（scikit-learn 0.22.2）来做到这一点：

import pandas
import sklearn
csv = pandas.read_csv('data.csv')
train, test = sklearn.model_selection.train_test_split(csv, train_size = 0.5)

Answer 5

为了回答@ desmond.carros问题，我将最佳答案修改如下，

 import random
 file=open("datafile.txt","r")
 data=list()
 for line in file:
    data.append(line.split(#your preferred delimiter))
 file.close()
 random.shuffle(data)
 train_data = data[:int((len(data)+1)*.80)] #Remaining 80% to training set
 test_data = data[int(len(data)*.80+1):] #Splits 20% data to test set

该代码将整个数据集拆分为80％的训练和20％的测试数据

Answer 6

sklearn.cross_validation从版本0.18开始不推荐使用,您应该使用sklearn.model_selection如下所示

from sklearn.model_selection import train_test_split
import numpy

with open("datafile.txt", "rb") as f:
   data = f.read().split('\n')
   data = numpy.array(data)  #convert array to numpy type array

   x_train ,x_test = train_test_split(data,test_size=0.5)       #test_size=0.5(whole_data)