使用Python MechanicalSoup登录时出现503错误

Question

我想在登录页面后面抓取一些信息，但得到503

当我尝试使用Mechanicalsoup登录时（与robobrowser的结果相同），将发生以下情况：

>>> import mechanicalsoup
>>> browser = mechanicalsoup.StatefulBrowser(user_agent='Mozilla/5.0')
>>> page = browser.get('https://X.com')
>>> page.status_code
200
>>> page = browser.get('https://X.com/wp-login.php')
>>> page.status_code
503

我尝试了几个不同的user_agents，如何解决呢？移动饼干？

Answer 1

好的，我设法使用https://github.com/Anorov/cloudflare-scrape做到了这一点

import cfscrape
from bs4 import BeautifulSoup

# log in
scraper = cfscrape.CloudflareScraper()
scraper.get('https://X.com/wp-login.php')
tokens = cfscrape.get_tokens('https://X.com')
browser = mechanicalsoup.StatefulBrowser(session=scraper, user_agent=tokens[1])
browser.select_form('#loginform')
browser['log'] = 'X'
browser['pwd'] = 'X'
browser.submit_selected()
browser.open('https://X.com/page/')