使用 spacy 进行 POS 模式挖掘

Question

我正在尝试使用 python 3 中的 spacy 从文本中提取语言特征。我的输入看起来像这样

Sent_id Text
1   I am exploring text analytics using spacy
2   amazing spacy is going to help me

我正在通过使用我提供的特定 POS 模式将单词提取为三元组/二元组短语来寻找这样的输出。像 NOUN VERB NOUN、ADJ NOUN 等，并保留数据帧结构。如果一个句子中有多个短语，则必须用新短语复制记录。

Sent_id Text    Feature Pattern
1   I am exploring text analytics using spacy   exploring text analytics    VERB NOUN NOUN
1   I am exploring text analytics using spacy   analytics using spacy   NOUN VERB NOUN
2   amazing spacy is going to help me   amazing spacy   ADJ NOUN

Answer 1

代码解释在评论里

import spacy
import pandas as pd
import re

# Load spacy model once and reuse 
nlp = spacy.load('en_core_web_sm')

# The dataframe with text
df = pd.DataFrame({
        'Sent_id': [1,2],
        'Text': [ "I am exploring text analytics using spacy", "amazing spacy is going to help me"]
    }) 

# Patters we are intrested in 
patterns = ["VERB NOUN", "NOUN VERB NOUN"]

# Convert each pattern into regular expression
re_patterns = [" ".join(["(\w+)_!"+pos for pos in p.split()]) for p in patterns]


def extract(nlp, text, patterns, re_patterns):
    """Extracts the pieces in text maching the POS pattern in patterns

    args:
        nlp : Loaded Spicy model object
        text: The input text
        patterns: The list of patters to be searched
        re_patterns: The patterns converted into regex

    returns: A list of tuples of form (t,p) where 
    t is the part of text matching the pattern p in patterns
    """
    doc = nlp(text)   
    matches = list()
    text_pos = " ".join([token.text+"_!"+token.pos_ for token in doc])
    for i, pattern in enumerate(re_patterns):
        for result in re.findall(pattern, text_pos):
            matches.append([" ".join(result), patterns[i]])
    return matches

# Test it 
print (extract(nlp, "A sleeping cat and walking dog", patterns, re_patterns))
# Returns
# [['sleeping cat', 'VERB NOUN'], ['walking dog', 'VERB NOUN']]

# Extract the matched patterns
df['matches'] = df['Text'].apply(lambda x: extract(nlp,x,patterns,re_patterns))


# Convert the list of tuples into rows
df = df.matches.apply(pd.Series).merge(df, right_index = True, left_index = True).drop(["matches"], axis = 1)\
.melt(id_vars = ['Sent_id', 'Text'], value_name = "matches").drop("variable", axis = 1)

# Add the matched text and matched patterns into new columns
df[['matched_text','matched_pattern']]= df.matches.apply(pd.Series)

# Drop the column and cleanup
df = df.drop("matches", axis = 1).sort_values('Sent_id')
df = df.drop_duplicates(subset =["matched_text", "matched_pattern"], keep='last') 

输出：

    Sent_id     Text                                matched_text    matched_pattern
0   1   I am exploring text analytics using spacy   exploring text  VERB NOUN
2   1   I am exploring text analytics using spacy   using spacy     VERB NOUN
4   1   I am exploring text analytics using spacy   analytics using spacy   NOUN VERB NOUN
1   2   amazing spacy is going to help me           NaN              NaN