使用BeautifulSoup/Python解析目录中的每个文件,另存为新文件

Question

Python和BeautifulSoup的新手.我有一个Python程序打开一个名为"example.html"的文件,在其上运行一个BeautifulSoup操作,然后在其上运行Bleach操作,然后将结果保存为文件"example-packaged.html".到目前为止,它正在为"example.html"的所有内容工作.

我需要修改它,以便打开文件夹"/ posts /"中的每个文件,在其上运行程序,然后将其保存为"/posts-cleaned/X-cleaned.html",其中X是原始文件名.

这是我的代码,最小化:

from bs4 import BeautifulSoup
import bleach
import re

text = BeautifulSoup(open("posts/example.html"))
text.encode("utf-8")

tag_black_list = ['iframe', 'script']
tag_white_list = ['p','div']
attr_white_list = {'*': ['title']}

# Step one, with BeautifulSoup: Remove tags in tag_black_list, destroy contents.
[s.decompose() for s in text(tag_black_list)]
pretty = (text.prettify())

# Step two, with Bleach: Remove tags and attributes not in whitelists, leave tag contents.
cleaned = bleach.clean(pretty, strip="TRUE", attributes=attr_white_list, tags=tag_white_list)

fout = open("posts/example-cleaned.html", "w")
fout.write(cleaned.encode("utf-8"))
fout.close()
print "Done"

很高兴收到现有解决方案的帮助和指示!

Answer 1

您可以使用os.listdir()获取目录中所有文件的列表。如果要在目录树中一直向下递归，则需要os.walk().

我会将所有这些代码移动到处理单个文件的功能，然后编写第二个函数来处理解析整个目录。像这样的东西：

def clean_dir(directory):

    os.chdir(directory)

    for filename in os.listdir(directory):
        clean_file(filename)

def clean_file(filename):

    tag_black_list = ['iframe', 'script']
    tag_white_list = ['p','div']
    attr_white_list = {'*': ['title']}

    with open(filename, 'r') as fhandle:
        text = BeautifulSoup(fhandle)
        text.encode("utf-8")

        # Step one, with BeautifulSoup: Remove tags in tag_black_list, destroy contents.
        [s.decompose() for s in text(tag_black_list)]
        pretty = (text.prettify())

        # Step two, with Bleach: Remove tags and attributes not in whitelists, leave tag contents.
        cleaned = bleach.clean(pretty, strip="TRUE", attributes=attr_white_list, tags=tag_white_list)

        # this appends -cleaned to the file; 
        # relies on the file having a '.'
        dot_pos = filename.rfind('.')
        cleaned_filename = '{0}-cleaned{1}'.format(filename[:dot_pos], filename[dot_pos:])

        with open(cleaned_filename, 'w') as fout:
            fout.write(cleaned.encode("utf-8"))

    print "Done"

然后你只需打电话clean_dir('/posts')或不打电话。

我将“-cleaned”附加到文件中，但我想我喜欢你更好地使用全新目录的想法。这样，如果-cleaned某个文件等已经存在，您就不必处理冲突。

我还使用该with语句在此处打开文件，因为它会关闭文件并自动处理异常。