BeautifulSoup - 在标签内按文字搜索

Question

请注意以下问题:

import re
from bs4 import BeautifulSoup as BS

soup = BS("""
<a href="/customer-menu/1/accounts/1/update">
    Edit
</a>
""")

# This returns the <a> element
soup.find(
    'a',
    href="/customer-menu/1/accounts/1/update",
    text=re.compile(".*Edit.*")
)

soup = BS("""
<a href="/customer-menu/1/accounts/1/update">
    <i class="fa fa-edit"></i> Edit
</a>
""")

# This returns None
soup.find(
    'a',
    href="/customer-menu/1/accounts/1/update",
    text=re.compile(".*Edit.*")
)

出于某种原因,当<i>标签存在时,BeautifulSoup将不匹配文本.查找标签并显示其文本

>>> a2 = soup.find(
        'a',
        href="/customer-menu/1/accounts/1/update"
    )
>>> print(repr(a2.text))
'\n Edit\n'

对.根据Docs,汤使用正则表达式的匹配函数,而不是搜索函数.所以我需要提供DOTALL标志:

pattern = re.compile('.*Edit.*')
pattern.match('\n Edit\n')  # Returns None

pattern = re.compile('.*Edit.*', flags=re.DOTALL)
pattern.match('\n Edit\n')  # Returns MatchObject

好的.看起来不错.我们来试试汤吧

soup = BS("""
<a href="/customer-menu/1/accounts/1/update">
    <i class="fa fa-edit"></i> Edit
</a>
""")

soup.find(
    'a',
    href="/customer-menu/1/accounts/1/update",
    text=re.compile(".*Edit.*", flags=re.DOTALL)
)  # Still return None... Why?!

编辑

我基于geckons的解决方案回答:我实现了这些帮助:

import re

MATCH_ALL = r'.*'


def like(string):
    """
    Return a compiled regular expression that matches the given
    string with any prefix and postfix, e.g. if string = "hello",
    the returned regex matches r".*hello.*"
    """
    string_ = string
    if not isinstance(string_, str):
        string_ = str(string_)
    regex = MATCH_ALL + re.escape(string_) + MATCH_ALL
    return re.compile(regex, flags=re.DOTALL)


def find_by_text(soup, text, tag, **kwargs):
    """
    Find the tag in soup that matches all provided kwargs, and contains the
    text.

    If no match is found, return None.
    If more than one match is found, raise ValueError.
    """
    elements = soup.find_all(tag, **kwargs)
    matches = []
    for element in elements:
        if element.find(text=like(text)):
            matches.append(element)
    if len(matches) > 1:
        raise ValueError("Too many matches:\n" + "\n".join(matches))
    elif len(matches) == 0:
        return None
    else:
        return matches[0]

现在,当我想找到上面的元素时,我就跑了 find_by_text(soup, 'Edit', 'a', href='/customer-menu/1/accounts/1/update')

Answer 1

问题是你<a>的<i>标签里面有标签,没有string你期望它拥有的属性.首先让我们来看看有什么text=""参数find().

注意:text参数是一个旧名称,因为它被称为BeautifulSoup 4.4.0 string.

来自文档:

尽管string用于查找字符串,但您可以将其与查找标记的参数结合使用:Beautiful Soup将查找其.string与您的字符串值匹配的所有标记.此代码查找.string为"Elsie"的标记:

soup.find_all("a", string="Elsie")
# [<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>]

现在,让我们来看看什么Tag的string(从属性是文档再次):

如果标记只有一个子节点,并且该子节点是NavigableString,则该子节点可用作.string:

title_tag.string
# u'The Dormouse's story'

(......)

如果一个标签包含多个东西,那么不清楚.string应该引用什么,所以.string被定义为None:

print(soup.html.string)
# None

这正是你的情况.您的<a>标记包含文本和 <i>标记.因此,None当尝试搜索字符串时,find会获得,因此无法匹配.

怎么解决这个？

也许有一个更好的解决方案,但我可能会这样:

import re
from bs4 import BeautifulSoup as BS

soup = BS("""
<a href="/customer-menu/1/accounts/1/update">
    <i class="fa fa-edit"></i> Edit
</a>
""")

links = soup.find_all('a', href="/customer-menu/1/accounts/1/update")

for link in links:
    if link.find(text=re.compile("Edit")):
        thelink = link
        break

print(thelink)

我认为没有太多的链接指向/customer-menu/1/accounts/1/update它应该足够快.

Answer 2

在一行中使用lambda

soup.find(lambda tag:tag.name=="a" and "Edit" in tag.text)

Answer 3

如果文本包含"编辑",则可以传递返回的函数Truea .find

In [51]: def Edit_in_text(tag):
   ....:     return tag.name == 'a' and 'Edit' in tag.text
   ....: 

In [52]: soup.find(Edit_in_text, href="/customer-menu/1/accounts/1/update")
Out[52]: 
<a href="/customer-menu/1/accounts/1/update">
<i class="fa fa-edit"></i> Edit
</a>

编辑:

您可以使用该.get_text()方法而不是text函数中的方法,该方法提供相同的结果:

def Edit_in_text(tag):
    return tag.name == 'a' and 'Edit' in tag.get_text()

Answer 4

在soupsieve 2.1.0中，您可以使用:-soup-containscss 伪类选择器来定位节点的文本。这取代了已弃用的:contains().

from bs4 import BeautifulSoup as BS

soup = BS("""
<a href="/customer-menu/1/accounts/1/update">
    Edit
</a>
""")
single = soup.select_one('a:-soup-contains("Edit")').text.strip()
multiple = [i.text.strip() for i in soup.select('a:-soup-contains("Edit")')]
print(single, '\n', multiple)