在Python中从文本创建序列向量

Question

我现在正在尝试为基于LSTM的NN准备输入数据.我有一些大量的文本文档,我想要的是为每个文档制作序列向量,以便我能够将它们作为列车数据提供给LSTM RNN.

我糟糕的做法:

import re
import numpy as np
#raw data
train_docs = ['this is text number one', 'another text that i have']

#put all docs together
train_data = ''
for val in train_docs:
    train_data += ' ' + val

tokens = np.unique(re.findall('[a-z?-?0-9]+', train_data.lower()))
voc = {v: k for k, v in dict(enumerate(tokens)).items()}

然后brutforce用"voc"词典替换每个doc.

有没有可以帮助完成这项任务的库？

Answer 1

解决了Keras文本预处理类:http: //keras.io/preprocessing/text/

像这样做:

from keras.preprocessing.text import Tokenizer, text_to_word_sequence

train_docs = ['this is text number one', 'another text that i have']
tknzr = Tokenizer(lower=True, split=" ")
tknzr.fit_on_texts(train_docs)
#vocabulary:
print(tknzr.word_index)

Out[1]:
{'this': 2, 'is': 3, 'one': 4, 'another': 9, 'i': 5, 'that': 6, 'text': 1, 'number': 8, 'have': 7}

#making sequences:
X_train = tknzr.texts_to_sequences(train_docs)
print(X_train)

Out[2]:
[[2, 3, 1, 8, 4], [9, 1, 6, 5, 7]]