dha*_*bro 10 php curl web-crawler web-scraping
我无法在curl中设置主机.如果我使用以下代码,它仍显示为localhost
function wget($url)
{
$agent= 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1';
$curlHeaders = array (
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding: gzip, deflate',
'Accept-Language: en-US,en;q=0.5',
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0.1',
'Connection: Keep-Alive',
'Pragma: no-cache',
'Referer: http://example.com/',
'Host: hostname',
'Cache-Control: no-cache',
'Cookie: visid_incap_185989=9v1q8Ar0ToSOja48BRmb8nn1GFUAAAAAQUIPAAAAAABCRWagbDIfmlN9NTrcvrct; incap_ses_108_185989=Z1orY6Bd0z3nGYE2lbJ/AXn1GFUAAAAAmb41m+jMLFCJB1rTIF28Mg==; _ga=GA1.3.637468927.1427699070; _gat=1; frontend=rqg7g9hp2ht788l309m7gk8qi7; _gat_UA-1279175-12=1; __utma=233911437.637468927.1427699070.1427699078.1427699078.1; __utmb=233911437.2.10.1427699078; __utmc=233911437; __utmz=233911437.1427699078.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmt_UA-1279175-1=1; _cb_ls=1; _chartbeat2=S0WVXDwMWnCFBgQp.1427699081322.1427699232786.1; PRUM_EPISODES=s=1427699568560&r=http%3A//example.com/'
);
$ch = curl_init();
curl_setopt ($ch, CURLOPT_HTTPHEADER, $curlHeaders);
curl_setopt ($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
$result=curl_exec($ch);
return $result;
}
Run Code Online (Sandbox Code Playgroud)
我fiddler
用来跟踪网络请求.我发现主机仍然是localhost
如果我在浏览器中加载相同的链接,我会得到以下内容 fiddler
我需要访问我指定的域名.我怎样才能做到这一点?注意:我知道主机名不应包含协议.
另外
另外我想知道是否可以通过终端在浏览器中看到网站的源代码?
假设我们没有尝试欺骗Host
标题,则Host
完全省略标题并让 curl 对其进行排序。在这种情况下,只需删除,'Host: hostname',
因为您已经使用 curl 自动将其设置为靠近底部的代码curl_setopt($ch, CURLOPT_URL, $url);
。
如果你真的想自己设置Host
标题,那么只需替换
'Host: hostname',
Run Code Online (Sandbox Code Playgroud)
和
"Host: ". parse_url($url, PHP_URL_HOST),
(注意:此函数不适用于相对 URL。)
尝试这样,
curl_init('XXX.XXX.XXX.XXX');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
Run Code Online (Sandbox Code Playgroud)