带有身份验证的urllib.request.urlopen(url)

Question

我一直在玩美丽的汤和解析网页几天.我一直在使用一行代码,这些代码在我编写的所有脚本中都是我的救星.代码行是:

r = requests.get('some_url', auth=('my_username', 'my_password')).

但......

我想用(打开一个带有身份验证的URL)做同样的事情:

(1) sauce = urllib.request.urlopen(url).read() (1)
(2) soup = bs.BeautifulSoup(sauce,"html.parser") (2)

我无法打开网址并阅读需要身份验证的网页.我如何实现这样的目标:

  (3) sauce = urllib.request.urlopen(url, auth=(username, password)).read() (3) 
instead of (1)

Answer 1

你正在使用HTTP Basic Authentication:

import urllib2, base64

request = urllib2.Request(url)
base64string = base64.b64encode('%s:%s' % (username, password))
request.add_header("Authorization", "Basic %s" % base64string)   
result = urllib2.urlopen(request)

因此,您应该对用户名和密码进行base64编码并将其作为base64标头发送.

Answer 2

从官方文档中查看HOWTO使用urllib包获取Internet资源:

# create a password manager
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler = urllib.request.HTTPBasicAuthHandler(password_mgr)

# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(handler)

# use the opener to fetch a URL
opener.open(a_url)

# Install the opener.
# Now all calls to urllib.request.urlopen use our opener.
urllib.request.install_opener(opener)

Answer 3

使用urllib3 \xe2\x80\xaf：

\n

import urllib3\n\nhttp = urllib3.PoolManager()\nmyHeaders = urllib3.util.make_headers(basic_auth=\'my_username:my_password\')\nhttp.request(\'GET\', \'http://example.org\', headers=myHeaders)\n

\n