Keras文本预处理 - 将Tokenizer对象保存到文件以进行评分

Question

我通过以下步骤(广泛地)使用Keras库训练了情绪分类器模型.

1. 使用Tokenizer对象/类将Text语料库转换为序列
2. 使用model.fit()方法构建模型
3. 评估此模型

现在,使用此模型进行评分,我能够将模型保存到文件并从文件加载.但是我没有找到将Tokenizer对象保存到文件的方法.如果没有这个,我每次需要得到一个句子时都必须处理语料库.有没有解决的办法？

Answer 1

最常见的方法是使用pickle或joblib.这里有一个如何使用pickle以保存的示例Tokenizer:

import pickle

# saving
with open('tokenizer.pickle', 'wb') as handle:
    pickle.dump(tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

# loading
with open('tokenizer.pickle', 'rb') as handle:
    tokenizer = pickle.load(handle)

Answer 2

Tokenizer类具有将日期保存为JSON格式的功能：

tokenizer_json = tokenizer.to_json()
with io.open('tokenizer.json', 'w', encoding='utf-8') as f:
    f.write(json.dumps(tokenizer_json, ensure_ascii=False))

可以使用以下tokenizer_from_json函数加载数据keras_preprocessing.text：

with open('tokenizer.json') as f:
    data = json.load(f)
    tokenizer = tokenizer_from_json(data)

Answer 3

接受的答案清楚地演示了如何保存标记器.以下是关于(一般)拟合或保存后得分问题的评论.假设一个列表texts包含两个列表的Train_text和Test_text,其中一组在令牌Test_text是集令牌的子集Train_text(一种乐观的假设).然后,fit_on_texts(Train_text)给出了不同的结果texts_to_sequences(Test_text)与第一次相比,通话fit_on_texts(texts)后text_to_sequences(Test_text).

具体例子:

from keras.preprocessing.text import Tokenizer

docs = ["A heart that",
         "full up like",
         "a landfill",
        "no surprises",
        "and no alarms"
         "a job that slowly"
         "Bruises that",
         "You look so",
         "tired happy",
         "no alarms",
        "and no surprises"]
docs_train = docs[:7]
docs_test = docs[7:]
# EXPERIMENT 1: FIT  TOKENIZER ONLY ON TRAIN
T_1 = Tokenizer()
T_1.fit_on_texts(docs_train)  # only train set
encoded_train_1 = T_1.texts_to_sequences(docs_train)
encoded_test_1 = T_1.texts_to_sequences(docs_test)
print("result for test 1:\n%s" %(encoded_test_1,))

# EXPERIMENT 2: FIT TOKENIZER ON BOTH TRAIN + TEST
T_2 = Tokenizer()
T_2.fit_on_texts(docs)  # both train and test set
encoded_train_2 = T_2.texts_to_sequences(docs_train)
encoded_test_2 = T_2.texts_to_sequences(docs_test)
print("result for test 2:\n%s" %(encoded_test_2,))

结果:

result for test 1:
[[3], [10, 3, 9]]
result for test 2:
[[1, 19], [5, 1, 4]]

当然,如果不满足上述乐观假设并且Test_text中的令牌集与Train_test的令牌不相交,则测试1会产生一个空括号列表 [].