测试和训练数据集具有不同数量的特征

Question

我正在尝试在一些训练和测试数据上训练svm模型.如果我将测试和训练数据结合起来,程序运行良好,但是如果我将它们分开并测试它所说的模型精度

Traceback (most recent call last):
  File "/home/PycharmProjects/analysis.py", line 160, in <module>
  main()
  File "/home/PycharmProjects/analysis.py", line 156, in main
  learn_model(tf_idf_train,target,tf_idf_test)
  File "/home/PycharmProjects/analysis.py", line 113, in learn_model
  predicted = classifier.predict(data_test)
  File "/home/.local/lib/python3.4/site-packages/sklearn/svm/base.py", line 573, in predict
  y = super(BaseSVC, self).predict(X)
  File "/home/.local/lib/python3.4/site-packages/sklearn/svm/base.py", line 310, in predict
  X = self._validate_for_predict(X)
  File "/home/.local/lib/python3.4/site-packages/sklearn/svm/base.py", line 479, in _validate_for_predict
  (n_features, self.shape_fit_[1]))
  ValueError: X.shape[1] = 19137 should be equal to 4888, the number of features at training time

这里的测试集大于列车集.因此测试集自然具有比trainset更多的特征.因此它给出值错误.

这是我的代码:

def load_train_file():
  with open('~1k comments.csv',encoding='ISO-8859-1',) as csv_file:
  reader = csv.reader(csv_file,delimiter=",",quotechar='"')
  reader.__next__()
  data =[]
  target = []
  for row in reader:
  if row[0] and row[1]:
  data.append(row[0])
  target.append(row[1])

  return data,target


  def load_file():
  with open('comments.csv',encoding='ISO-8859-1',) as csv_file:
  reader = csv.reader(csv_file,delimiter=",",quotechar='"')
  reader.__next__()
  data =[]
  target = []
  for row in reader:
  if row[0] and row[1]:
  data.append(row[0])
  target.append(row[1])
  print(len(data))

  return data

  # preprocess creates the term frequency matrix for the review data set
  def preprocess():
  dataTrain,targetTrain = load_train_file()
  testData=load_file()
  count_vectorizer = CountVectorizer(binary='true')
  dataTrain = count_vectorizer.fit_transform(dataTrain)
  tfidf_train_data = TfidfTransformer(use_idf=True).fit_transform(dataTrain)

  count_vectorizer = CountVectorizer()
  testData = count_vectorizer.fit_transform(testData)
  tfidf_test_data = TfidfTransformer(use_idf=True).fit_transform(testData)

  return tfidf_train_data,tfidf_test_data

  def learn_model(data,target,testData):
  data_train,data_test,target_train,target_test = cross_validation.train_test_split(data,target,test_size=0.001,random_state=43)
  e = np.zeros(testData.shape[0])
  data_train1, data_test, target_train1, target_test = cross_validation.train_test_split(testData, e,test_size=.9,random_state=43)
  classifier = SVC(gamma=.01, C=100.)
  classifier.fit(data_train, target_train)
  predicted = classifier.predict(data_test)
  for x in range(0,50):
  print(testData[x]+str(predicted[x]))

  def evaluate_model(target_true,target_predicted):
  print (classification_report(target_true,target_predicted))
  print ("The accuracy score is {:.2%}".format(accuracy_score(target_true,target_predicted)))

  def main():
  data,target = load_train_file()
  datatest=load_file()


  tf_idf_train,tf_idf_test = preprocess()
  # print(tf_idf_train.shape())
  # print(tf_idf_test.shape())

  learn_model(tf_idf_train,target,tf_idf_test)
  # learn_model(data,target,datatest)


  main()

怎么解决这个？

Answer 1

对于火车和测试部件,必须使用相同的矢量器和变压器; 另外,矢量化器不应该适合测试数据.而不是

count_vectorizer = CountVectorizer(binary='true')
dataTrain = count_vectorizer.fit_transform(dataTrain)
tfidf_train_data = TfidfTransformer(use_idf=True).fit_transform(dataTrain)

count_vectorizer = CountVectorizer()
testData = count_vectorizer.fit_transform(testData)
tfidf_test_data = TfidfTransformer(use_idf=True).fit_transform(testData)

使用这样的东西:

count_vectorizer = CountVectorizer(binary=True)
tfidf_transformer = TfidfTransformer(use_idf=True) 
dataTrain = count_vectorizer.fit_transform(dataTrain)
tfidf_train_data = transformer.fit_transform(dataTrain)

testData = count_vectorizer.transform(testData)
tfidf_test_data = tfidf_transformer.transform(testData)

您还可以使用Pipeline使其更好:

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(
    CountVectorizer(binary=True),
    TfidfTransformer(use_idf=True),
)
tfidf_train_data = pipe.fit_transform(dataTrain) 
tfidf_test_data = pipe.transform(testData)

或者甚至使用TfidfVectorizer,它将CountVectorizer和TfidfTransformer组合在一个矢量化器对象中:

from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer(binary=True, use_idf=True)
tfidf_train_data = vec.fit_transform(dataTrain) 
tfidf_test_data = vec.transform(testData)