我需要从.aspx网页上抓取查询结果.
http://legistar.council.nyc.gov/Legislation.aspx
网址是静态的,那么如何向此网页提交查询并获得结果？假设我们需要从相应的下拉菜单中选择"所有年份"和"所有类型".
那里的人必须知道如何做到这一点.
我在https://searchlight.cluen.com/E5/CandidateSearch.aspx上有一个ASPX页面,上面有一个表单,我想提交并解析信息.
使用Python的urllib和urllib2我使用正确的标头和用户代理创建了一个post请求.但是生成的html响应不包含预期的结果表.我误解了还是错过了任何明显的细节？
import urllib
import urllib2
headers = {
'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13',
'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml; q=0.9,*/*; q=0.8',
'Content-Type': 'application/x-www-form-urlencoded'
}
# obtained these values from viewing the source of https://searchlight.cluen.com/E5/CandidateSearch.aspx
viewstate = '/wEPDwULLTE3NTc4MzQwNDIPZBYCAg ... uJRWDs/6Ks1FECco='
eventvalidation = '/wEWjQMC8pat6g4C77jgxg0CzoqI8wgC3uWinQQCwr/ ... oPKYVeb74='
url = 'https://searchlight.cluen.com/E5/CandidateSearch.aspx'
formData = (
('__VIEWSTATE', viewstate),
('__EVENTVALIDATION', eventvalidation),
('__EVENTTARGET',''),
('__EVENTARGUMENT',''),
('textcity',''),
('dropdownlistposition',''),
('dropdownlistdepartment',''),
('dropdownlistorderby',''),
('textsearch',''),
)
# change user agent
from urllib import FancyURLopener
class MyOpener(FancyURLopener):
version = 'Mozilla/5.0 …我想从" http://www.indiapost.gov.in/pin/ " 废弃PINCODE ,我正在编写以下代码.
import urllib
import urllib2
headers = {
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Origin': 'http://www.indiapost.gov.in',
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17',
'Content-Type': 'application/x-www-form-urlencoded',
'Referer': 'http://www.indiapost.gov.in/pin/',
'Accept-Encoding': 'gzip,deflate,sdch',
'Accept-Language': 'en-US,en;q=0.8',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
}
viewstate = 'JulXDv576ZUXoVOwThQQj4bDuseXWDCZMP0tt+HYkdHOVPbx++G8yMISvTybsnQlNN76EX/...'
eventvalidation = '8xJw9GG8LMh6A/b6/jOWr970cQCHEj95/6ezvXAqkQ/C1At06MdFIy7+iyzh7813e1/3Elx...'
url = 'http://www.indiapost.gov.in/pin/'
formData = (
('__EVENTVALIDATION', eventvalidation),
('__EVENTTARGET',''),
('__EVENTARGUMENT',''),
('__VIEWSTATE', viewstate),
('__VIEWSTATEENCRYPTED',''),
('__EVENTVALIDATION', eventvalidation),
('txt_offname',''),
('ddl_dist','0'),
('txt_dist_on',''),
('ddl_state','2'),
('btn_state','Search'),
('txt_stateon',''),
('hdn_tabchoice','3')
)
from urllib import FancyURLopener
class MyOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, …我尝试使用Python中的Beautifulsoup解析50 000个url.解析在循环中工作:
我发现解析一个页面的时间是15-18秒.从第一页开始,我会抓住20个元素.
为什么Beautifulsoup工作这么慢？如何在Python中加速Beautifulsoup？