读取文本文件并将其拆分为python中的单个单词

Question

所以我有这个文本文件由数字和单词组成,例如像这样 - 09807754 18 n 03 aristocrat 0 blue_blood 0 patrician我想拆分它,以便每个单词或数字都会作为一个新行出现.

一个空白分隔符是理想的,因为我希望带有破折号的单词保持连接.

这是我到目前为止:

f = open('words.txt', 'r')
for word in f:
    print(word)

我不确定如何离开这里,我希望这是输出:

09807754
18
n
3
aristocrat
...

Answer 1

如果您的数据周围没有引号,并且您一次只想要一个单词(忽略文件中空格与换行符的含义):

with open('words.txt','r') as f:
    for line in f:
        for word in line.split():
           print(word)      

如果需要文件每行中单词的嵌套列表(例如,要从文件创建行和列的矩阵):

with open("words.txt") as f:
    [line.split() for line in f]

或者,如果要将文件展平为文件中单个单词的平面列表,则可以执行以下操作:

with open('words.txt') as f:
    [word for line in f for word in line.split()]

如果你想要一个正则表达式解决方案:

import re
with open("words.txt") as f:
    for line in f:
        for word in re.findall(r'\w+', line):
            # word by word

或者,如果您希望它是带有正则表达式的逐行生成器:

 with open("words.txt") as f:
     (word for line in f for word in re.findall(r'\w+', line))

Answer 2

f = open('words.txt')
for word in f.read().split():
    print(word)

Answer 3

作为补充,如果您正在阅读vvvvery大文件,并且您不希望一次将所有内容读入内存,您可以考虑使用缓冲区,然后按yield返回每个单词:

def read_words(inputfile):
    with open(inputfile, 'r') as f:
        while True:
            buf = f.read(10240)
            if not buf:
                break

            # make sure we end on a space (word boundary)
            while not str.isspace(buf[-1]):
                ch = f.read(1)
                if not ch:
                    break
                buf += ch

            words = buf.split()
            for word in words:
                yield word
        yield '' #handle the scene that the file is empty

if __name__ == "__main__":
    for word in read_words('./very_large_file.txt'):
        process(word)

Answer 4

你可以做的是使用 nltk 来标记单词，然后将所有单词存储在一个列表中，这就是我所做的。如果你不知道 nltk; 它代表自然语言工具包，用于处理自然语言。如果你想开始，这里有一些资源 [ http://www.nltk.org/book/]

import nltk 
from nltk.tokenize import word_tokenize 
file = open("abc.txt",newline='')
result = file.read()
words = word_tokenize(result)
for i in words:
       print(i)

输出将是这样的：

09807754
18
n
03
aristocrat
0
blue_blood
0
patrician