通过HTTP上传BIG文件

Question

我正在尝试使用PowerShell将非常大的VM映像(5-15 Gb大小)上传到HTTP服务器.

我尝试使用那几个方法(这里链接到net.WebClient.UploadFile 脚本和脚本与Invoke-webRequest)

它适用于小于2GB的文件,但不适用于大于此值的文件.

我正在尝试直接使用httpWebRequest但我无法投入FileStream其中.

所以我的问题是:如何将文件流放入webrequest？

或者更一般地说:如何使用PowerShell通过http上传大文件？

$Timeout=10000000;
$fileName = "0.iso";
$data = "C:\\$fileName";
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName";
#$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": 
#$buffer = gc -en byte $data # too much space in memory 
$buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb
[System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
#$webRequest.ContentLength = $buffer.Length;
$webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123");

$requestStream = $webRequest.GetRequestStream()
$requestStream.Write($buffer, 0, $buffer.Length)
$requestStream.Flush()
$requestStream.Close()

[System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse()
$streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream())
$result = $streamReader.ReadToEnd()
return $result
$stream.Close() 

Answer 1

默认情况下,HttpWebRequest正在缓冲内存中的数据.只需将HttpWebRequest.AllowWriteStreamBuffering属性设置为false,您就可以上传几乎任何大小的文件.在msdn上查看更多详细信息

Answer 2

谢谢@Stoune,最后一件事有助于获得最终的工作解决方案.

另外,需要组织流文件读取和写入webrequest缓冲区.它可能与那段代码有关:

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

最终的脚本看起来像这样:

$user = "admin"
$pass = "admin123"
$dir = "C:\Virtual Hard Disks"
$fileName = "win2012r2std.vhdx"
$file = "$dir/$fileName"
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"
$Timeout=10000000
$bufSize=10000

$cred = New-Object System.Net.NetworkCredential($user, $pass)

$webRequest = [System.Net.HttpWebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
$webRequest.AllowWriteStreamBuffering=$false
$webRequest.SendChunked=$true # needed by previous line
$webRequest.Credentials = $cred

$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
  while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
  {
    $requestStream.write($chunk, 0, $bytesRead)
    $requestStream.Flush()
  }

$responceStream = $webRequest.getresponse()
#$status = $webRequest.statuscode

$FileStream.Close()
$requestStream.Close()
$responceStream.Close()

$responceStream
$responceStream.GetResponseHeader("Content-Length") 
$responceStream.StatusCode
#$status