提取 YouTube 搜索结果时出现问题

Question

我是 Python 新手，并且已经学习了 bs4 的 Web Scraping 基础知识。在这里我尝试提取 Youtube 搜索结果的所有链接，它不像其他网站那样工作。我分析了搜索结果html数据，搜索结果的链接位于id为“video title”的锚标记中，但该标记没有出现在我的bs4解析的html文档中

from bs4 import BeautifulSoup as bs
import requests
name=input("Enter video name ")
url='https://www.youtube.com/results?search_query='+name
searched=requests.get(url)
soup=bs(searched.text,'html.parser')
aid=soup.find_all('a',{'id':'video-title'})
print(aid)

我希望输出包含所有搜索结果。我还没有学过其他软件包，如果可能的话，我想在 bs4 中做到这一点。

Answer 1

所有这些获取 YouTube 搜索结果数据的麻烦只是浪费时间和精力。

为什么不尝试这些选项

• YouTube 数据 API
• 硒+无头镀铬

也就是说，对于前 20 个结果，您可以从源中的 JavaScript 内容获取数据。下面给出了答案。

经过大约 1 小时理解生成的 json 后，某些查询仍然失败。YouTube 是一个非常复杂的网站。响应可能会因位置、浏览器、搜索查询等而异。

我们正在从源中的此脚本标记中提取数据。

代码：

from bs4 import BeautifulSoup as bs
import requests
import re
import json
headers={
'User-Agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36'
}
name=input("Enter video name: ")
url='https://www.youtube.com/results?search_query=hello'+name
searched=requests.get(url,headers=headers)
soup=bs(searched.text,'html.parser')
aid=soup.find('script',string=re.compile('ytInitialData'))
extracted_josn_text=aid.text.split(';')[0].replace('window["ytInitialData"] =','').strip()
video_results=json.loads(extracted_josn_text)
#print(item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][1])
item_section=video_results["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"][0]["itemSectionRenderer"]["contents"]

for item in item_section:
    try:
        video_info=item["videoRenderer"]
        title=video_info["title"]["simpleText"]
        url=video_info["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"]
        print('Title:',title)
        print('Url:',url, end="\n----------\n")
    except KeyError:
            pass

输出：

Enter video name: hello
Title: New Punjabi Songs 2017-Hello Hello(Ful Song)-Prince Narula-Yuvika Chaudhary-Latest Punjabi Song 2017
Url: /watch?v=mv326-zVpAQ
----------
Title: Alan Walker - The Spectre
Url: /watch?v=wJnBTPUQS5A
----------
Title: Hello Hello (Full HD) - Rajvir Jawanda | MixSingh | Josan Bros | New Punjabi Songs 2018
Url: /watch?v=xydupjQSj44
----------
Title: Bachchan - Hello Hello - Kannada Movie Full Song Video | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=oLMMgoug4Uk
----------
Title: Hello Hello latest 2017 16 june punjabi song
Url: /watch?v=MqCSsPXw8QU
----------
Title:   Hello Hello   | + More Kids Songs | Super Simple Songs
Url: /watch?v=saDkICxEdgY
----------
Title: 'Gallan Goodiyaan' Full VIDEO Song | Dil Dhadakne Do | T-Series
Url: /watch?v=jCEdTq3j-0U
----------
Title: Hello Hello Gippy Grewal Feat. Dr. Zeus Full Song HD | Latest Punjabi Song 2013
Url: /watch?v=IRW2O4QZhgs
----------
Title: Hello Hello | Pataakha | Malaika Arora | Vishal Bhardwaj & Rekha Bhardwaj | Gulzar | Ganesh Acharya
Url: /watch?v=RxBAitQLSLA
----------
Title: Hello Hello (Lyrical Audio) Prince Narula ft. Yuvika Chaudhary | Punjabi Lyrical Audio 2017 | WHM
Url: /watch?v=v8VIsIvhDoQ
----------
Title: Hello Hello Full Video Song || Bhale Bhale Magadivoi || Nani, Lavanya Tripathi
Url: /watch?v=y3FI02OO_kU
----------
Title: Hello hello gaad bahe dhufee na egaa  (new comedy hhhhhh)
Url: /watch?v=DuRrcTo4rgg
----------
Title: Proper Patola - Official Video | Namaste England | Arjun | Parineeti | Badshah | Diljit | Aastha
Url: /watch?v=YmXJp4RtBCM
----------
Title: Official Video: Nikle Currant Song | Jassi Gill | Neha Kakkar | Sukh-E Muzical Doctorz | Jaani
Url: /watch?v=uBaqgt5V0mU
----------
Title: Insane (Full Song)  Sukhe - Jaani - Arvindr Khaira - White Hill Music - Latest Punjabi Song 2018
Url: /watch?v=mKpPhVVF8So
----------
Title: Radha bole HELLO HELLO-cartoon song mix with step up 2
Url: /watch?v=TFCTgNCzrck
----------
Title: Hello Song | CoCoMelon Nursery Rhymes & Kids Songs
Url: /watch?v=fxVMqaViVaA
----------
Title: Bachchan - Hello Hello Unplugged Version | Sudeep | Bhavana | V Harikrishna
Url: /watch?v=lvH3kTGJeEQ
----------
Title: Hello Hello! Can You Clap Your Hands? | Original  Kids Song | Super Simple Songs
Url: /watch?v=fN1Cyr0ZK9M
----------

您可以尝试的最后一件事是模拟 youtube 本身使用的 API

IE。将请求发送至

https://www.youtube.com/results?search_query=yoursearchtext

它有很多 cookie 和会话值作为参数发送。您可能需要模拟所有这些。您可能需要使用Requests 会话对象来执行此操作。