使用Selenium/PhantomJS进行网络捕获

Question

我想使用Selenium和python捕获我正在浏览的网站的流量,因为使用代理的流量将是https不会让我走得太远.

我的想法是运行带有selenium的phantomJS并使用phantomJS来执行脚本(不是在页面上使用webdriver.execute_script(),而是在phantomJS本身上).我在考虑netlog.js脚本(来自https://github.com/ariya/phantomjs/blob/master/examples/netlog.js).

因为它在命令行中这样工作

phantomjs --cookies-file=/tmp/foo netlog.js https://google.com

用硒必须有类似的方法吗？

提前致谢

更新:

用browsermob-proxy解决了它.

pip3 install browsermob-proxy

Python3代码

from selenium import webdriver
from browsermobproxy import Server

server = Server(<path to browsermob-proxy>)
server.start()
proxy = server.create_proxy({'captureHeaders': True, 'captureContent': True, 'captureBinaryContent': True})

service_args = ["--proxy=%s" % proxy.proxy, '--ignore-ssl-errors=yes']
driver = webdriver.PhantomJS(service_args=service_args)

proxy.new_har()
driver.get('https://google.com')
print(proxy.har)  # this is the archive
# for example:
all_requests = [entry['request']['url'] for entry in proxy.har['log']['entries']]

Answer 1

如果这里有人正在寻找纯 Selenium/Python 解决方案，以下代码片段可能会有所帮助。它使用 Chrome 记录所有请求，并作为示例打印所有 json 请求及其相应的响应。

from time import sleep

from selenium import webdriver
from selenium.webdriver import DesiredCapabilities

# make chrome log requests
capabilities = DesiredCapabilities.CHROME

capabilities["loggingPrefs"] = {"performance": "ALL"}  # chromedriver < ~75
# capabilities["goog:loggingPrefs"] = {"performance": "ALL"}  # chromedriver 75+

driver = webdriver.Chrome(
    desired_capabilities=capabilities, executable_path="./chromedriver"
)

# fetch a site that does xhr requests
driver.get("https://sitewithajaxorsomething.com")
sleep(5)  # wait for the requests to take place

# extract requests from logs
logs_raw = driver.get_log("performance")
logs = [json.loads(lr["message"])["message"] for lr in logs_raw]

def log_filter(log_):
    return (
        # is an actual response
        log_["method"] == "Network.responseReceived"
        # and json
        and "json" in log_["params"]["response"]["mimeType"]
    )

for log in filter(log_filter, logs):
    request_id = log["params"]["requestId"]
    resp_url = log["params"]["response"]["url"]
    print(f"Caught {resp_url}")
    print(driver.execute_cdp_cmd("Network.getResponseBody", {"requestId": request_id}))

要点：https://gist.github.com/lorey/079c5e178c9c9d3c30ad87df7f70491d

Answer 2

我正在使用代理

from selenium import webdriver
from browsermobproxy import Server

server = Server(environment.b_mob_proxy_path)
server.start()
proxy = server.create_proxy()
service_args = ["--proxy-server=%s" % proxy.proxy]
driver = webdriver.PhantomJS(service_args=service_args)

proxy.new_har()
driver.get('url_to_open')
print proxy.har  # this is the archive
# for example:
all_requests = [entry['request']['url'] for entry in proxy.har['log']['entries']]

'har'(http存档格式)有很多关于请求和响应的其他信息,对我来说非常有用

在Linux上安装:

pip install browsermob-proxy