是否可以在HttpClient中解压缩之前访问压缩数据？

Question

是否可以在HttpClient中解压缩之前访问压缩数据？

我正在研究Google Cloud Storage .NET客户端库.有三个功能(在.NET,我的客户端库和存储服务之间)以不愉快的方式组合:

下载文件(Google云端存储术语中的对象)时,服务器会包含存储数据的哈希值.然后,我的客户端代码根据下载的数据验证该哈希值.
Google云端存储的一个独立功能是,当请求包含匹配的Accept-Encoding时,用户可以设置对象的Content-Encoding,并在下载时将其作为标题包含在内.(目前,当请求不包括那个时,让我们忽略行为......)
HttpClientHandler 可以自动,透明地解压缩gzip(或deflate)内容.

当所有这三个组合在一起时,我们就会遇到麻烦.这是一个简短但完整的程序,演示了这一点,但没有使用我的客户端库(并点击可公开访问的文件):

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.GZip
        };
        var client = new HttpClient(handler);

        var response = await client.GetAsync(url);
        byte[] content = await response.Content.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");

        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");

        using (var md5 = MD5.Create())
        {
            var md5Hash = md5.ComputeHash(content);
            var md5HashBase64 = Convert.ToBase64String(md5Hash);
            Console.WriteLine($"MD5 of content: {md5HashBase64}");
        }
    }
}

Run Code Online (Sandbox Code Playgroud)

.NET Core项目文件:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <LangVersion>7.1</LangVersion>
  </PropertyGroup>
</Project>

Run Code Online (Sandbox Code Playgroud)

输出:

Content: hello world
Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
MD5 of content: XrY7u+Ae7tCTyyK7j1rNww==

Run Code Online (Sandbox Code Playgroud)

如您所见,内容的MD5与X-Goog-Hash标题的MD5部分不同.(在我的客户端库中,我使用的是crc32c哈希,但是显示了相同的行为.)

这不是一个错误HttpClientHandler- 它是预期的,但是当我想验证哈希时会很痛苦.基本上,我需要在内容前和解压后.而且我找不到任何办法.

在一定程度上澄清我的要求,我知道如何防止在减压HttpClient和从流中读取的时候,而不是事后解压缩-但我需要能够做到这一点不改变任何使用所产生的代码HttpResponseMessage从HttpClient.(有很多代码可以处理响应,我只想在一个中心位置进行更改.)

我有一个计划,我已经制作了原型,并且到目前为止已经发现了它,但是有点难看.它涉及创建一个三层处理程序:

HttpClientHandler 禁用自动解压缩.
一个新的处理程序,它用一个新的Stream子类替换内容流,该子类委托给原始内容流,但在读取时散列数据.
基于Microsoft DecompressionHandler代码的仅解压缩处理程序.

虽然这有效,但它有以下缺点:

开源许可:根据MIT许可的Microsoft代码,检查我需要做什么才能在我的仓库中创建新文件
有效地分叉MS代码,这意味着我应该定期检查以查看是否有任何错误
Microsoft代码使用程序集的内部成员,因此它不会像它可能一样干净地移植.

如果微软DecompressionHandler上市,那将会有很大帮助 - 但这可能会比我需要的时间更长.

如果可能的话,我正在寻找的是另一种方法 - 我错过了一些让我在解压缩之前得到内容的东西.我不想重新发明HttpClient- 例如,响应经常被分解,我不想进入那方面.这是我正在寻找的一个非常具体的拦截点.

Answer 1

shm*_*lie 12

看看@Michael给我的暗示我失踪了.得到压缩的内容,你可以使用后CryptoStream,和GZipStream,和StreamReader没有加载到内存超过需要读取响应.CryptoStream将解压缩和读取时压缩内容.替换为StreamReadera FileStream,您可以将数据写入具有最小内存使用量的文件:)

using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.None
        };
        var client = new HttpClient(handler);
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

        var response = await client.GetAsync(url);
        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        string text = null;
        using (var md5 = MD5.Create())
        {
            using (var cryptoStream = new CryptoStream(await response.Content.ReadAsStreamAsync(), md5, CryptoStreamMode.Read))
            {
                using (var gzipStream = new GZipStream(cryptoStream, CompressionMode.Decompress))
                {
                    using (var streamReader = new StreamReader(gzipStream, Encoding.UTF8))
                    {
                        text = streamReader.ReadToEnd();
                    }
                }
                Console.WriteLine($"Content: {text}");
                var md5HashBase64 = Convert.ToBase64String(md5.Hash);
                Console.WriteLine($"MD5 of content: {md5HashBase64}");
            }
        }
    }
}

Run Code Online (Sandbox Code Playgroud)

输出:

Hash header: crc32c=T1s5RQ==,md5=xhF4M6pNFRDQnvaRRNVnkA==
Content: hello world
MD5 of content: xhF4M6pNFRDQnvaRRNVnkA==

Run Code Online (Sandbox Code Playgroud)

答案的V2

在阅读Jon的回复和更新的答案后,我有以下版本.几乎相同的想法,但我把流媒体转移到HttpContent我注入的特殊.不完全漂亮,但想法就在那里.

using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        string url = "https://www.googleapis.com/download/storage/v1/b/"
            + "storage-library-test-bucket/o/gzipped-text.txt?alt=media";
        var handler = new HttpClientHandler
        {
            AutomaticDecompression = DecompressionMethods.None
        };
        var client = new HttpClient(new Intercepter(handler));
        client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

        var response = await client.GetAsync(url);
        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        HttpContent content1 = response.Content;
        byte[] content = await content1.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");
        var md5Hash = ((HashingContent)content1).Hash;
        var md5HashBase64 = Convert.ToBase64String(md5Hash);
        Console.WriteLine($"MD5 of content: {md5HashBase64}");
    }

    public class Intercepter : DelegatingHandler
    {
        public Intercepter(HttpMessageHandler innerHandler) : base(innerHandler)
        {
        }

        protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
        {
            var response = await base.SendAsync(request, cancellationToken);
            response.Content = new HashingContent(await response.Content.ReadAsStreamAsync());
            return response;
        }
    }

    public sealed class HashingContent : HttpContent
    {
        private readonly StreamContent streamContent;
        private readonly MD5 mD5;
        private readonly CryptoStream cryptoStream;
        private readonly GZipStream gZipStream;

        public HashingContent(Stream content)
        {
            mD5 = MD5.Create();
            cryptoStream = new CryptoStream(content, mD5, CryptoStreamMode.Read);
            gZipStream = new GZipStream(cryptoStream, CompressionMode.Decompress);
            streamContent = new StreamContent(gZipStream);
        }

        protected override Task SerializeToStreamAsync(Stream stream, TransportContext context) => streamContent.CopyToAsync(stream, context);
        protected override bool TryComputeLength(out long length)
        {
            length = 0;
            return false;
        }

        protected override Task<Stream> CreateContentReadStreamAsync() => streamContent.ReadAsStreamAsync();

        protected override void Dispose(bool disposing)
        {
            try
            {
                if (disposing)
                {
                    streamContent.Dispose();
                    gZipStream.Dispose();
                    cryptoStream.Dispose();
                    mD5.Dispose();
                }
            }
            finally
            {
                base.Dispose(disposing);
            }
        }

        public byte[] Hash => mD5.Hash;
    }
}

Run Code Online (Sandbox Code Playgroud)

Answer 2

Ale*_*Hgs 5

我设法通过以下方式获得正确的 headerhash：

创建继承 HttpClientHandler 的自定义处理程序
压倒性的SendAsync
使用字节读取响应base.SendAsync
使用 GZipStream 压缩它
将 Gzip Md5 哈希为 base64（使用您的代码）

这个问题是，正如你所说，“减压之前”在这里并没有得到真正的尊重

我们的想法是让这个if工作如您所愿 https://github.com/dotnet/corefx/blob/master/src/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpResponseParser.cs#L80 -L91

它匹配

class Program
{
    const string url = "https://www.googleapis.com/download/storage/v1/b/storage-library-test-bucket/o/gzipped-text.txt?alt=media";

    static async Task Main()
    {
        //await HashResponseContent(CreateHandler(DecompressionMethods.None));
        //await HashResponseContent(CreateHandler(DecompressionMethods.GZip));
        await HashResponseContent(new MyHandler());

        Console.ReadLine();
    }

    private static HttpClientHandler CreateHandler(DecompressionMethods decompressionMethods)
    {
        return new HttpClientHandler { AutomaticDecompression = decompressionMethods };
    }

    public static async Task HashResponseContent(HttpClientHandler handler)
    {
        //Console.WriteLine($"Using AutomaticDecompression : '{handler.AutomaticDecompression}'");
        //Console.WriteLine($"Using SupportsAutomaticDecompression : '{handler.SupportsAutomaticDecompression}'");
        //Console.WriteLine($"Using Properties : '{string.Join('\n', handler.Properties.Keys.ToArray())}'");

        var client = new HttpClient(handler);

        var response = await client.GetAsync(url);
        byte[] content = await response.Content.ReadAsByteArrayAsync();
        string text = Encoding.UTF8.GetString(content);
        Console.WriteLine($"Content: {text}");

        var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
        Console.WriteLine($"Hash header: {hashHeader}");
        byteArrayToMd5(content);

        Console.WriteLine($"=====================================================================");
    }

    public static string byteArrayToMd5(byte[] content)
    {
        using (var md5 = MD5.Create())
        {
            var md5Hash = md5.ComputeHash(content);
            return Convert.ToBase64String(md5Hash);
        }
    }

    public static byte[] Compress(byte[] contentToGzip)
    {
        using (MemoryStream resultStream = new MemoryStream())
        {
            using (MemoryStream contentStreamToGzip = new MemoryStream(contentToGzip))
            {
                using (GZipStream compressionStream = new GZipStream(resultStream, CompressionMode.Compress))
                {
                    contentStreamToGzip.CopyTo(compressionStream);
                }
            }

            return resultStream.ToArray();
        }
    }
}

public class MyHandler : HttpClientHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);
        var responseContent = await response.Content.ReadAsByteArrayAsync().ConfigureAwait(false);

        Program.byteArrayToMd5(responseContent);

        var compressedResponse = Program.Compress(responseContent);
        var compressedResponseMd5 = Program.byteArrayToMd5(compressedResponse);

        Console.WriteLine($"recompressed response to md5 : {compressedResponseMd5}");

        return response;
    }
}

Run Code Online (Sandbox Code Playgroud)

这之所以有效，是因为我“碰巧”使用 .NET 使用默认设置对该文件的内容进行 gzip 压缩。但是有几种不同的方法来压缩内容，最终会创建不同的哈希值。如果 gzip 稳定（即压缩相同的输入总是给出相同的输出），这将是可行的 - 但它不适用于这种情况:( (4认同)

Answer 3

Mic*_*ael 5

Accept-Encoding禁用自动解压，手动添加标头，然后在哈希验证后解压怎么样？

private static async Task Test2()
{
    var url = @"https://www.googleapis.com/download/storage/v1/b/storage-library-test-bucket/o/gzipped-text.txt?alt=media";
    var handler = new HttpClientHandler
    {
        AutomaticDecompression = DecompressionMethods.None
    };
    var client = new HttpClient(handler);
    client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip");

    var response = await client.GetAsync(url);
    var raw = await response.Content.ReadAsByteArrayAsync();

    var hashHeader = response.Headers.GetValues("X-Goog-Hash").FirstOrDefault();
    Debug.WriteLine($"Hash header: {hashHeader}");

    bool match = false;
    using (var md5 = MD5.Create())
    {
        var md5Hash = md5.ComputeHash(raw);
        var md5HashBase64 = Convert.ToBase64String(md5Hash);
        match = hashHeader.EndsWith(md5HashBase64);
        Debug.WriteLine($"MD5 of content: {md5HashBase64}");
    }

    if (match)
    {
        var memInput = new MemoryStream(raw);
        var gz = new GZipStream(memInput, CompressionMode.Decompress);
        var memOutput = new MemoryStream();
        gz.CopyTo(memOutput);
        var text = Encoding.UTF8.GetString(memOutput.ToArray());
        Console.WriteLine($"Content: {text}");
    }
}

Run Code Online (Sandbox Code Playgroud)

如果我们谈论的是千兆字节，那么这个方法就无法使用，抱歉:( (3认同)
这基本上是我的原型的一个更简单但效率较低的版本。问题在于，当这些文件可能有多个千兆字节时，它将整个流保留在内存中。我需要在从内容返回的流中插入哈希值:( (2认同)

归档时间：	8 年前
查看次数：	4167 次
最近记录：	8 年前