Python,无法获取嵌入式视频网址

Question

好吧,我一直在摸不着头脑.我正在尝试使用Beautiful Soup在网页上检索嵌入视频的URL,并在Python 2.7.6中请求模块.我检查了chrome中的html,我可以看到视频的网址,但是当我使用请求获取页面并使用Beautiful Soup时,我找不到"视频"节点.从查看源代码看,视频窗口看起来像是一个嵌套的html文档.我已经搜遍了所有,无法找出为什么我无法检索这个.如果有人能指出我正确的方向,我会非常感激.谢谢.

这是其中一个视频的网址:

http://www.growingagreenerworld.com/episode125/

Answer 1

问题是,有一个iframe与video标签内,其在浏览器中异步加载.

好消息是,您可以通过向iframeURL传递当前页面URL 的附加请求来模拟该行为Referer.

执行:

import re

from bs4 import BeautifulSoup
import requests

url = 'http://www.growingagreenerworld.com/episode125/'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}

with requests.Session() as session:
    session.headers = headers

    response = session.get(url)

    soup = BeautifulSoup(response.content)

    # follow the iframe url
    response = session.get('http:' + soup.iframe['src'], headers={'Referer': url})
    soup = BeautifulSoup(response.content)

    # extract the video URL from the script tag
    print re.search(r'"url":"(.*?)"', soup.script.text).group(1)

打印:

http://pdl.vimeocdn.com/43109/378/290982236.mp4?token2=1424891659_69f846779e96814be83194ac3fc8fbae&aksessionid=678424d1f375137f