卷曲:传输关闭,剩余未完成的读取数据

Question

卷曲:传输关闭,剩余未完成的读取数据

我遇到了大卷毛问题.

我明白了

nread <= 0,服务器关闭连接,bailing

传输关闭,剩余未完成的读数据

并且内容部分传递

    GET /stats/?stats_breakdown=track__track&campaign=&search_criteria=2&period=0&date_month=11&date_day=03&date_year=2015&start_date_month=11&start_date_day=03&start_date_year=2015&end_date_month=12&end_date_day=31&end_date_year=2014 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13
Host: domain.com
Accept: */*
Cookie: sessionid=xxg4gglsm7o3b224wihqz8od19wl31h1; csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C
Cache-Control: no-cache
Connection: Keep-Alive
Keep-Alive: 600
Accept-Language: en-us
X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR
Content-Type: application/x-www-form-urlencoded
Origin: https://domain.com
Referer: https://domain.com

* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* additional stuff not fine transfer.c:1037: 0 0
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Server: nginx/1.8.0
< Date: Wed, 04 Nov 2015 12:54:05 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Vary: Cookie, Accept-Language
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"
< Content-Language: en
* Replaced cookie csrftoken="JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C" for domain domain.com, path /, expire 1478091245
< Set-Cookie: csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C; expires=Wed, 02-Nov-2016 12:54:05 GMT; Max-Age=31449600; Path=/; secure
<
* nread <= 0, server closed connection, bailing
* transfer closed with outstanding read data remaining
* Closing connection #0

Run Code Online (Sandbox Code Playgroud)

这是我使用的php配置

function getHeaders()
{
    $headers = array();
    $headers[] = 'Cache-Control: no-cache';
    $headers[] = 'Connection: Keep-Alive';
    $headers[] = 'Keep-Alive: 600';
    $headers[] = 'Accept-Language: en-us';
    $headers[] = 'X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR';

    $headers[] = 'Content-Type: application/x-www-form-urlencoded';
    $headers[] = 'Origin: https://domain.com';
    $headers[] = 'Referer: https://domain.com';

    return $headers;

}

curl_setopt($connection, CURLOPT_URL, $url);
    curl_setopt($connection, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($connection, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($connection, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
    curl_setopt($connection, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($connection, CURLOPT_SSL_VERIFYHOST, false);
    curl_setopt($connection, CURLOPT_COOKIEFILE, 'cookie.txt');
    curl_setopt($connection, CURLOPT_COOKIEJAR, 'cookie.txt');
    curl_setopt($connection, CURLOPT_CONNECTTIMEOUT ,550000000);
    curl_setopt($connection, CURLOPT_TIMEOUT, 5500000000); //timeout in seconds
    curl_setopt($connection, CURLOPT_HTTPHEADER, getHeaders());
    curl_setopt($connection, CURLOPT_VERBOSE, 1);

Run Code Online (Sandbox Code Playgroud)

Answer 1

W. *_*ckx 8

好的,经过一些搜索和IRC聊天后我找到了解决方案,但并不是100%确定原因是什么.看起来保持活动不足以保持连接继续.将在此处发布解决方案,希望我可以帮助某人.

对我有帮助的是补充

--keepalive-time 2

Run Code Online (Sandbox Code Playgroud)

卷曲选项的解释

--keepalive-time <seconds>

Run Code Online (Sandbox Code Playgroud)

此选项设置连接在发送keepalive探测之前需要保持空闲的时间以及各个keepalive探测之间的时间.它目前在提供TCP_KEEPIDLE和TCP_KEEPINTVL套接字选项(即Linux,最近的AIX,HP-UX等)的操作系统上有效.如果使用--no-keepalive,则此选项无效.(在7.18.0中添加)

如果多次使用此选项,将使用最后一个选项.如果未指定,则选项默认为60秒.

看起来默认值太高,无法保持连接打开.

这是我用来打电话的完整命令

curl URL -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,et;q=0.6,nl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' --compressed -v --keepalive-time 2

Run Code Online (Sandbox Code Playgroud)

而且我在osx上运行这个版本的curl

curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

Run Code Online (Sandbox Code Playgroud)

如果有人想在PHP curl中使用此选项,则自PHP 5.5起可以使用--keepalive-time选项.您可以按如下方式使用它:

curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1);
curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2);

Run Code Online (Sandbox Code Playgroud)

希望这有助于有人在同样的问题上挣扎!

Answer 2

Hem*_*nth 6

Encountered similar issue, my server is behind nginx. Curl is able to receive a response if it connects to the server directly but if curl connects to the server via nginx, curl is throwing below error

Session
* transfer closed with outstanding read data remaining
* Closing connection 0 curl: (18) transfer closed with outstanding read data remaining

When I connect to same nginx URL using a browser, Response is showing up fine. It is very weird, When I tried to connect to the same nginx URL using curl, it was throwing above error.

After comparing headers sent by the browser and curl. I found that the browser is able to receive the response because of the below header which curl is not sending:

'Accept-Encoding: gzip'

Run Code Online (Sandbox Code Playgroud)

Sending above header using curl is working fine. So, what the above header is doing is compressing response to gzip thus reducing the response size.

After some more digging, found that nginx is not able to send any payload > 80kb. After wasting a lot of time, found that issue is with nginx buffering and nginx worked like a charm after adding below proxy_buffering property in nginx.conf:

 location / {
 proxy_buffering off;
 }

Run Code Online (Sandbox Code Playgroud)

The accepted answer didn't resolve my issue. Writing this answer, so that anyone need not waste their time if they are facing the same issue as I did.

归档时间：	10 年，2 月前
查看次数：	20403 次
最近记录：	6 年，3 月前