如何使用pandas从一个数据框创建测试和训练样本？

Question

我有一个相当大的数据集形式的数据集,我想知道如何将数据帧分成两个随机样本(80%和20%)进行训练和测试.

谢谢!

Answer 1

scikit learntrain_test_split是一个很好的.

from sklearn.model_selection import train_test_split

train, test = train_test_split(df, test_size=0.2)

Answer 2

我会用numpy的randn:

In [11]: df = pd.DataFrame(np.random.randn(100, 2))

In [12]: msk = np.random.rand(len(df)) < 0.8

In [13]: train = df[msk]

In [14]: test = df[~msk]

只是为了看到这个有效:

In [15]: len(test)
Out[15]: 21

In [16]: len(train)
Out[16]: 79

Answer 3

熊猫随机样本也会起作用

train=df.sample(frac=0.8,random_state=200) #random state is a seed value
test=df.drop(train.index)

Answer 4

我会使用scikit-learn自己的training_test_split,并从索引中生成它

from sklearn.cross_validation import train_test_split


y = df.pop('output')
X = df

X_train,X_test,y_train,y_test = train_test_split(X.index,y,test_size=0.2)
X.iloc[X_train] # return dataframe train

Answer 5

无需转换为 numpy。只需使用pandas df 进行拆分，它将返回一个pandas df。

from sklearn.model_selection import train_test_split

train, test = train_test_split(df, test_size=0.2)

如果你想从 y 中拆分 x

X_train, X_test, y_train, y_test = train_test_split(df[list_of_x_cols], df[y_col],test_size=0.2)

如果你想拆分整个 df

X, y = df[list_of_x_cols], df[y_col]

Answer 6

您可以使用以下代码创建测试和训练样本:

from sklearn.model_selection import train_test_split
trainingSet, testSet = train_test_split(df, test_size=0.2)

测试大小可能会根据您要在测试和训练数据集中放入的数据百分比而有所不同.

Answer 7

创建训练/测试甚至验证样本的方法有很多。

情况1：train_test_split没有任何选择的经典方式：

from sklearn.model_selection import train_test_split
train, test = train_test_split(df, test_size=0.3)

情况2：非常小的数据集（<500行）的情况：为了通过交叉验证获得所有行的结果。最后，您将对可用训练集的每一行都有一个预测。

from sklearn.model_selection import KFold
kf = KFold(n_splits=10, random_state=0)
y_hat_all = []
for train_index, test_index in kf.split(X, y):
    reg = RandomForestRegressor(n_estimators=50, random_state=0)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    clf = reg.fit(X_train, y_train)
    y_hat = clf.predict(X_test)
    y_hat_all.append(y_hat)

情况3a：用于分类目的的不平衡数据集。在第一种情况之后，这里是等效的解决方案：

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.3)

情况3b：出于分类目的的不平衡数据集。在案例2之后，这是等效的解决方案：

from sklearn.model_selection import StratifiedKFold
kf = StratifiedKFold(n_splits=10, random_state=0)
y_hat_all = []
for train_index, test_index in kf.split(X, y):
    reg = RandomForestRegressor(n_estimators=50, random_state=0)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]
    clf = reg.fit(X_train, y_train)
    y_hat = clf.predict(X_test)
    y_hat_all.append(y_hat)

案例4：您需要在大数据上创建训练/测试/验证集以调整超参数（训练60％，测试20％和验证20％）。

from sklearn.model_selection import train_test_split
X_train, X_test_val, y_train, y_test_val = train_test_split(X, y, test_size=0.6)
X_test, X_val, y_test, y_val = train_test_split(X_test_val, y_test_val, stratify=y, test_size=0.5)

Answer 8

有许多有效的答案.再添一个.来自sklearn.cross_validation import train_test_split

#gets a random 80% of the entire set
X_train = X.sample(frac=0.8, random_state=1)
#gets the left out portion of the dataset
X_test = X.loc[~df_model.index.isin(X_train.index)]

Answer 9

您也可以考虑将分层划分为训练和测试集.Startized division还会随机生成训练和测试集,但这样可以保留原始的比例.这使得训练和测试集更好地反映了原始数据集的属性.

import numpy as np  

def get_train_test_inds(y,train_proportion=0.7):
    '''Generates indices, making random stratified split into training set and testing sets
    with proportions train_proportion and (1-train_proportion) of initial sample.
    y is any iterable indicating classes of each observation in the sample.
    Initial proportions of classes inside training and 
    testing sets are preserved (stratified sampling).
    '''

    y=np.array(y)
    train_inds = np.zeros(len(y),dtype=bool)
    test_inds = np.zeros(len(y),dtype=bool)
    values = np.unique(y)
    for value in values:
        value_inds = np.nonzero(y==value)[0]
        np.random.shuffle(value_inds)
        n = int(train_proportion*len(value_inds))

        train_inds[value_inds[:n]]=True
        test_inds[value_inds[n:]]=True

    return train_inds,test_inds

df [train_inds]和df [test_inds]为您提供原始DataFrame df的训练和测试集.

Answer 10

只需从 df 中选择范围行，如下所示

row_count = df.shape[0]
split_point = int(row_count*1/5)
test_data, train_data = df[:split_point], df[split_point:]

Answer 11

如果您需要根据数据集中的标签列拆分数据，您可以使用以下命令：

def split_to_train_test(df, label_column, train_frac=0.8):
    train_df, test_df = pd.DataFrame(), pd.DataFrame()
    labels = df[label_column].unique()
    for lbl in labels:
        lbl_df = df[df[label_column] == lbl]
        lbl_train_df = lbl_df.sample(frac=train_frac)
        lbl_test_df = lbl_df.drop(lbl_train_df.index)
        print '\n%s:\n---------\ntotal:%d\ntrain_df:%d\ntest_df:%d' % (lbl, len(lbl_df), len(lbl_train_df), len(lbl_test_df))
        train_df = train_df.append(lbl_train_df)
        test_df = test_df.append(lbl_test_df)

    return train_df, test_df

并使用它：

train, test = split_to_train_test(data, 'class', 0.7)

如果你想控制分裂随机性或使用一些全局随机种子，你也可以传递 random_state 。

Answer 12

您可以使用 ~ （波浪号运算符）排除使用 df.sample() 采样的行，让熊猫单独处理索引的采样和过滤，以获得两组。

train_df = df.sample(frac=0.8, random_state=100)
test_df = df[~df.index.isin(train_df.index)]