为什么随机搜索显示比网格搜索更好的结果？

Question

我正在使用scikit-learn的RandomizedSearchCV函数.一些学术论文声称,与整个网格搜索相比,随机搜索可以提供"足够好"的结果,但可以节省大量时间.

令人惊讶的是,有一次,RandomizedSearchCV提供了比GridSearchCV更好的结果.我认为GridSearchCV是穷举的,所以结果必须比RandomizedSearchCV更好,假设他们搜索同一个网格.

对于相同的数据集和大多数相同的设置,GridsearchCV返回了以下结果:
最佳cv精度:0.7642857142857142
测试集得分:0.725
最佳参数:'C':0.02

RandomizedSearchCV返回以下结果:最佳cv准确度:0.7428571428571429
测试集得分:0.7333333333333333
最佳参数:'C':0.008

对我来说,0.733的测试分数优于0.725,并且RandomizedSearchCV的测试分数和训练分数之间的差异较小,据我所知,这意味着过度拟合.

那么为什么GridSearchCV会让我的结果更糟？

GridSearchCV代码:

def linear_SVC(x, y, param, kfold):
    param_grid = {'C':param}
    k = KFold(n_splits=kfold, shuffle=True, random_state=0)
    grid = GridSearchCV(LinearSVC(), param_grid=param_grid, cv=k, n_jobs=4, verbose=1)

    return grid.fit(x, y)

#high C means more chance of overfitting

start = timer()
param = [i/1000 for i in range(1,1000)]
param1 = [i for i in range(1,101)]
param.extend(param1)

#progress = progressbar.bar.ProgressBar()
clf = linear_SVC(x=x_train, y=y_train, param=param, kfold=3)

print('LinearSVC:')
print('Best cv accuracy: {}' .format(clf.best_score_))
print('Test set score:   {}' .format(clf.score(x_test, y_test)))
print('Best parameters:  {}' .format(clf.best_params_))
print()

duration = timer() - start
print('time to run: {}' .format(duration))

RandomizedSearchCV代码:

from sklearn.model_selection import RandomizedSearchCV

def Linear_SVC_Rand(x, y, param, kfold, n):
    param_grid = {'C':param}
    k = StratifiedKFold(n_splits=kfold, shuffle=True, random_state=0)
    randsearch = RandomizedSearchCV(LinearSVC(), param_distributions=param_grid, cv=k, n_jobs=4,
                                    verbose=1, n_iter=n)

    return randsearch.fit(x, y)

start = timer()
param = [i/1000 for i in range(1,1000)]
param1 = [i for i in range(1,101)]
param.extend(param1)

#progress = progressbar.bar.ProgressBar()
clf = Linear_SVC_Rand(x=x_train, y=y_train, param=param, kfold=3, n=100)

print('LinearSVC:')
print('Best cv accuracy: {}' .format(clf.best_score_))
print('Test set score:   {}' .format(clf.score(x_test, y_test)))
print('Best parameters:  {}' .format(clf.best_params_))
print()

duration = timer() - start
print('time to run: {}' .format(duration))

Answer 1

首先,尝试理解这一点:https: //stats.stackexchange.com/questions/49540/understanding-stratified-cross-validation

所以应该知道StratifiedKFold比KFold更好.

在GridSearchCV和RandomizedSearchCV中使用StratifiedKFold.并确保设置" shuffle = False"并且不要使用" random_state"参数.这样做,您正在使用的数据集不会被洗牌,因此每次训练时您的结果都不会改变.你可能得到你期望的.

GridSearchCV代码:

def linear_SVC(x, y, param, kfold):
    param_grid = {'C':param}
    k = StratifiedKFold(n_splits=kfold)
    grid = GridSearchCV(LinearSVC(), param_grid=param_grid, cv=k, n_jobs=4, verbose=1)

    return grid.fit(x, y)

RandomizedSearchCV代码:

def Linear_SVC_Rand(x, y, param, kfold, n):
    param_grid = {'C':param}
    k = StratifiedKFold(n_splits=kfold)
    randsearch = RandomizedSearchCV(LinearSVC(), param_distributions=param_grid, cv=k, n_jobs=4,
                                    verbose=1, n_iter=n)

    return randsearch.fit(x, y)