从Python字符串中删除标点符号

Question

我似乎有一个问题从Python中的字符串中删除标点符号.在这里,我给了一个文本文件(特别是Project Gutenberg的一本书)和一个停用词列表.我想返回10个最常用单词的字典.不幸的是,我在返回的字典中不断打嗝.

import sys
import collections
from string import punctuation
import operator

#should return a string without punctuation
def strip_punc(s):
    return ''.join(c for c in s if c not in punctuation)

def word_cloud(infile, stopwordsfile):

    wordcount = {}

    #Reads the stopwords into a list
    stopwords = [x.strip() for x in open(stopwordsfile, 'r').readlines()]


    #reads data from the text file into a list
    lines = []
    with open(infile) as f:
        lines = f.readlines()
        lines = [line.split() for line in lines]

    #does the wordcount
    for line in lines:
        for word in line:
            word = strip_punc(word).lower()
            if word not in stopwords:
                if word not in wordcount:
                    wordcount[word] = 1
                else:
                    wordcount[word] += 1

    #sorts the dictionary, grabs 10 most common words
    output = dict(sorted(wordcount.items(),
                  key=operator.itemgetter(1), reverse=True)[:10])

    print(output)


if __name__=='__main__':

    try:

        word_cloud(sys.argv[1], sys.argv[2])

    except Exception as e:

        print('An exception has occured:')
        print(e)
        print('Try running as python3 word_cloud.py <input-text> <stopwords>')

这将打印出来

{'said': 659, 'mr': 606, 'one': 418, '“i': 416, 'lorry': 322, 'upon': 288, 'will': 276, 'defarge': 268, 'man': 264, 'little': 263}

"我不应该在那里.我不明白为什么在我的助手功能中没有消除它.

提前致谢.

Answer 1

角色“不是".

string.punctuation 仅包含以下ASCII字符:

In [1]: import string

In [2]: string.punctuation
Out[2]: '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

所以你需要扩充你正在剥离的字符列表.

像下面这样的东西应该完成你需要的东西:

extended_punc = punctuation + '“' #  and any other characters you need to strip

def strip_punc(s):
    return ''.join(c for c in s if c not in extended_punc)

或者,您可以使用该包unidecode对您的文本进行ASCII编码,而不必担心创建您可能需要处理的unicode字符列表:

from unidecode import unidecode

def strip_punc(s):
    s = unidecode(s.decode('utf-8'))
    return ''.join(c for c in s if c not in punctuation).encode('utf-8')