那些遇到过的问题

解压缩HTTPInputStream时GZIPInputStream过早关闭

Jef*_*ell 4 java amazon-s3 gzipinputstream

题

在下面的“编辑”部分中查看更新的问题

我正在尝试使用GZIPInputStream从Amazon S3快速解压缩大的（〜300M）GZIP文件，但它仅输出文件的一部分；但是，如果我在解压缩之前下载到文件系统，则GZIPInputStream将解压缩整个文件。

如何获得GZIPInputStream解压缩整个HTTPInputStream而不只是它的第一部分？

我尝试过的

请参阅下面的编辑部分中的更新

我怀疑有一个HTTP问题，只是没有抛出任何异常，GZIPInputStream每次都返回一个相当一致的文件块，据我所知，它总是在WET记录边界上中断，尽管每个选择的边界都是不同的URL（这很奇怪，因为所有内容都被视为二进制流，根本没有对文件中的WET记录进行任何解析。）

我可以找到的最接近的问题 是，从s3读取时GZIPInputStream被过早关闭。该问题的答案是，某些GZIP文件实际上是多个附加的GZIP文件，而GZIPInputStream处理得不好。但是，如果是这种情况，为什么GZIPInputStream在文件的本地副本上可以正常工作？

演示代码和输出

下面是一段示例代码，演示了我所遇到的问题。我已经在两个不同网络上的两台不同Linux计算机上使用Java 1.8.0_72和1.8.0_112对它进行了测试，结果相似。我希望来自解压缩的HTTPInputStream的字节数与来自文件的解压缩的本地副本的字节数相同，但是经过解压缩的HTTPInputStream小得多。

输出量 
Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 87894 bytes from HTTP->GZIP
Read 448974935 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile0.wet
------
Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00040-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 1772936 bytes from HTTP->GZIP
Read 451171329 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile40.wet
------
Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698541142.66/wet/CC-MAIN-20161202170901-00500-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 89217 bytes from HTTP->GZIP
Read 453183600 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile500.wet
Run Code Online (Sandbox Code Playgroud) 样例代码 
import java.net.*;
import java.io.*;
import java.util.zip.GZIPInputStream;
import java.nio.channels.*;

public class GZIPTest {
    public static void main(String[] args) throws Exception {
        // Our three test files from CommonCrawl
        URL url0 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz");
        URL url40 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00040-ip-10-31-129-80.ec2.internal.warc.wet.gz");
        URL url500 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698541142.66/wet/CC-MAIN-20161202170901-00500-ip-10-31-129-80.ec2.internal.warc.wet.gz");

        /*
         * Test the URLs and display the results
         */
        test(url0, "testfile0.wet");
        System.out.println("------");
        test(url40, "testfile40.wet");
        System.out.println("------");
        test(url500, "testfile500.wet");
    }

    public static void test(URL url, String testGZFileName) throws Exception {
        System.out.println("Testing URL "+url.toString());

        // First directly wrap the HTTPInputStream with GZIPInputStream
        // and count the number of bytes we read
        // Go ahead and save the extracted stream to a file for further inspection
        System.out.println("Testing HTTP Input Stream direct to GZIPInputStream");
        int bytesFromGZIPDirect = 0;
        URLConnection urlConnection = url.openConnection();
        FileOutputStream directGZIPOutStream = new FileOutputStream("./"+testGZFileName);

        // FIRST TEST - Decompress from HTTPInputStream
        GZIPInputStream gzipishttp = new GZIPInputStream(urlConnection.getInputStream());

        byte[] buffer = new byte[1024];
        int bytesRead = -1;
        while ((bytesRead = gzipishttp.read(buffer, 0, 1024)) != -1) {
            bytesFromGZIPDirect += bytesRead;
            directGZIPOutStream.write(buffer, 0, bytesRead); // save to file for further inspection
        }
        gzipishttp.close();
        directGZIPOutStream.close();

        // Now save the GZIPed file locally
        System.out.println("Testing saving to file before decompression");
        int bytesFromGZIPFile = 0;
        ReadableByteChannel rbc = Channels.newChannel(url.openStream());
        FileOutputStream outputStream = new FileOutputStream("./test.wet.gz");
        outputStream.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
        outputStream.close();

        // SECOND TEST - decompress from FileInputStream
        GZIPInputStream gzipis = new GZIPInputStream(new FileInputStream("./test.wet.gz"));

        buffer = new byte[1024];
        bytesRead = -1;
        while((bytesRead = gzipis.read(buffer, 0, 1024)) != -1) {
            bytesFromGZIPFile += bytesRead;
        }
        gzipis.close();

        // The Results - these numbers should match but they don't
        System.out.println("Read "+bytesFromGZIPDirect+" bytes from HTTP->GZIP");
        System.out.println("Read "+bytesFromGZIPFile+" bytes from HTTP->file->GZIP");
        System.out.println("Output from HTTP->GZIP saved to file "+testGZFileName);
    }

}
Run Code Online (Sandbox Code Playgroud)

编辑

根据@VGR的注释，演示代码中的Closed Stream和关联的Channel。

更新：

该问题似乎确实是文件特有的。我在本地（wget）下拉了Common Crawl WET存档，将其解压缩（gunzip 1.8），然后将其重新压缩（gzip 1.8），然后重新上传到S3，然后进行实时解压缩就可以了。如果您修改上面的示例代码以包括以下行，则可以看到测试：

// Original file from CommonCrawl hosted on S3
URL originals3 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz");
// Recompressed file hosted on S3
URL rezippeds3 = new URL("https://s3-us-west-1.amazonaws.com/com.jeffharwell.commoncrawl.gziptestbucket/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz");

test(originals3, "originalhost.txt");
test(rezippeds3, "rezippedhost.txt");
Run Code Online (Sandbox Code Playgroud)

URL rezippeds3指向我下载，解压缩和重新压缩，然后重新上传到S3的WET存档文件。您将看到以下输出：

Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 7212400 bytes from HTTP->GZIP
Read 448974935 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file originals3.txt
-----
Testing URL https://s3-us-west-1.amazonaws.com/com.jeffharwell.commoncrawl.gziptestbucket/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 448974935 bytes from HTTP->GZIP
Read 448974935 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file rezippeds3.txt
Run Code Online (Sandbox Code Playgroud)

如您所见，一旦文件被重新压缩，我就可以通过GZIPInputStream对其进行流传输并获取整个文件。原始文件仍显示减压的通常过早结束。当我下载并上传WET文件而不进行重新压缩时，我得到了同样的不完整流传输行为，因此绝对是通过重新压缩解决了该问题。我还将原始和重新压缩的文件都放到了传统的Apache Web服务器上，并且能够复制结果，因此S3似乎与问题无关。

所以。我有一个新问题。

新问题

在读取相同内容时，为什么FileInputStream的行为与HTTPInputStream不同。如果是完全相同的文件，为什么：

新的GZIPInputStream（urlConnection.getInputStream（））;

表现与

新的GZIPInputStream（新的FileInputStream（“ ./ test.wet.gz”））;

?? 输入流不只是输入流吗？

Jef*_*ell 5

根本原因讨论

事实证明InputStreams可以有很大的不同。特别是，它们在实现.available（）方法的方式上有所不同。例如，ByteArrayInputStream .available（）返回InputStream中剩余的字节数。但是，HTTPInputStream .available（）返回需要进行阻塞的IO请求以重新填充缓冲区之前可读取的字节数。（有关更多信息，请参见Java Docs）

问题在于，GZIPInputStream使用.available（）的输出来确定InputStream完成解压缩完整的GZIP文件后，是否在InputStream中可能存在其他GZIP文件。这是来自OpenJDK源文件GZIPInputStream.java方法readTrailer（）的231行。

   if (this.in.available() > 0 || n > 26) {
Run Code Online (Sandbox Code Playgroud)

如果HTTPInputStream读取缓冲区在两个串联的GZIP文件的边界处清空，则GZIPInputStream调用.available（），它将以0响应，因为它将需要进入网络以重新填充缓冲区，因此GZIPInputStream将文件视为完整并过早关闭。

Common Crawl .wet归档文件是数百兆字节的小型串联GZIP文件，因此，最终HTTPInputStream缓冲区将在串联的GZIP文件之一的末尾清空，并且GZIPInputStream会过早关闭。这解释了问题中显示的问题。

解决方案和解决方法

此GIST包含一个针对jdk8u152-b00修订版12039的补丁程序和两个jtreg测试，这些测试消除了（我认为很不正确）对.available（）的不正确依赖。

如果无法修补JDK，则要确保可用（）始终返回> 0，这将强制GZIPInputStream始终检查流中的另一个GZIP文件。不幸的是，HTTPInputStream是私有的，因此您不能直接将其子类化，而是扩展InputStream并包装HTTPInputStream。下面的代码演示了解决方法。

演示代码和输出

这里的输出显示，当按照所讨论的方式包装HTTPInputStream时，从文件直接从HTTP读取串联的GZIP时，GZIPInputStream将产生相同的结果。

Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 448974935 bytes from HTTP->GZIP
Read 448974935 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile0.wet
------
Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00040-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 451171329 bytes from HTTP->GZIP
Read 451171329 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile40.wet
------
Testing URL https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698541142.66/wet/CC-MAIN-20161202170901-00500-ip-10-31-129-80.ec2.internal.warc.wet.gz
Testing HTTP Input Stream direct to GZIPInputStream
Testing saving to file before decompression
Read 453183600 bytes from HTTP->GZIP
Read 453183600 bytes from HTTP->file->GZIP
Output from HTTP->GZIP saved to file testfile500.wet
Run Code Online (Sandbox Code Playgroud)

这是使用InputStream包装器修改的问题的演示代码。

import java.net.*;
import java.io.*;
import java.util.zip.GZIPInputStream;
import java.nio.channels.*;

public class GZIPTest {
    // Here is a wrapper class that wraps an InputStream
    // but always returns > 0 when .available() is called.
    // This will cause GZIPInputStream to always make another 
    // call to the InputStream to check for an additional 
    // concatenated GZIP file in the stream.
    public static class AvailableInputStream extends InputStream {
        private InputStream is;

        AvailableInputStream(InputStream inputstream) {
            is = inputstream;
        }

        public int read() throws IOException {
            return(is.read());
        }

        public int read(byte[] b) throws IOException {
            return(is.read(b));
        }

        public int read(byte[] b, int off, int len) throws IOException {
            return(is.read(b, off, len));
        }

        public void close() throws IOException {
            is.close();
        }

        public int available() throws IOException {
            // Always say that we have 1 more byte in the
            // buffer, even when we don't
            int a = is.available();
            if (a == 0) {
                return(1);
            } else {
                return(a);
            }
        }
    }



    public static void main(String[] args) throws Exception {
        // Our three test files from CommonCrawl
        URL url0 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00009-ip-10-31-129-80.ec2.internal.warc.wet.gz");
        URL url40 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698540409.8/wet/CC-MAIN-20161202170900-00040-ip-10-31-129-80.ec2.internal.warc.wet.gz");
        URL url500 = new URL("https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2016-50/segments/1480698541142.66/wet/CC-MAIN-20161202170901-00500-ip-10-31-129-80.ec2.internal.warc.wet.gz");

        /*
         * Test the URLs and display the results
         */
        test(url0, "testfile0.wet");
        System.out.println("------");
        test(url40, "testfile40.wet");
        System.out.println("------");
        test(url500, "testfile500.wet");
    }

    public static void test(URL url, String testGZFileName) throws Exception {
        System.out.println("Testing URL "+url.toString());

        // First directly wrap the HTTP inputStream with GZIPInputStream
        // and count the number of bytes we read
        // Go ahead and save the extracted stream to a file for further inspection
        System.out.println("Testing HTTP Input Stream direct to GZIPInputStream");
        int bytesFromGZIPDirect = 0;
        URLConnection urlConnection = url.openConnection();
        // Wrap the HTTPInputStream in our AvailableHttpInputStream
        AvailableInputStream ais = new AvailableInputStream(urlConnection.getInputStream());
        GZIPInputStream gzipishttp = new GZIPInputStream(ais);
        FileOutputStream directGZIPOutStream = new FileOutputStream("./"+testGZFileName);
        int buffersize = 1024;
        byte[] buffer = new byte[buffersize];
        int bytesRead = -1;
        while ((bytesRead = gzipishttp.read(buffer, 0, buffersize)) != -1) {
            bytesFromGZIPDirect += bytesRead;
            directGZIPOutStream.write(buffer, 0, bytesRead); // save to file for further inspection
        }
        gzipishttp.close();
        directGZIPOutStream.close();

        // Save the GZIPed file locally
        System.out.println("Testing saving to file before decompression");
        ReadableByteChannel rbc = Channels.newChannel(url.openStream());
        FileOutputStream outputStream = new FileOutputStream("./test.wet.gz");
        outputStream.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);

        // Now decompress the local file and count the number of bytes
        int bytesFromGZIPFile = 0;
        GZIPInputStream gzipis = new GZIPInputStream(new FileInputStream("./test.wet.gz"));

        buffer = new byte[1024];
        while((bytesRead = gzipis.read(buffer, 0, 1024)) != -1) {
            bytesFromGZIPFile += bytesRead;
        }
        gzipis.close();

        // The Results
        System.out.println("Read "+bytesFromGZIPDirect+" bytes from HTTP->GZIP");
        System.out.println("Read "+bytesFromGZIPFile+" bytes from HTTP->file->GZIP");
        System.out.println("Output from HTTP->GZIP saved to file "+testGZFileName);
    }

}
Run Code Online (Sandbox Code Playgroud)

  • 我将补丁提交给 OpenJDK core-libs-dev 小组供考虑。如果您想知道是否有任何更新，请检查错误 [JDK-8081450](https://bugs.openjdk.java.net/browse/JDK-8081450)。 (2认同)

归档时间：

查看次数：

547 次

最近记录：

8 年，11 月 前
相关归档
Android中的SQLite如何更新特定行 131
使用Java 8的Monads 75
如何在java中使用org.json.JSONObject将值设置为null？ 67
Android意图上下文令人困惑 54
运行ant时"无法找到tools.jar" 48
无法使用可执行文件C:\ Program Files\Java\jdk-10.0.1\bin\java.exe确定Java版本 48
如何在 IntelliJ 中修复“不受支持的类文件主要版本 60”？ 46
ConcurrentHashMap如何在内部工作？ 43
如何使用GraphQL将图像上传到AWS S3？ 9
AWS S3事件 - 客户端标识 1
难疑归档
如何查看远程Git分支？ 6408
在'for'循环中访问索引？ 3312
我怎么知道通过jQuery选择了哪个单选按钮？ 2583
为什么"使用命名空间std"被认为是不好的做法？ 2486
JavaScript等效于printf/String.Format 1874
为什么Java有瞬态字段？ 1406
如何在Python中打印到stderr？ 1246
如何使用jQuery设置/取消设置cookie？ 1209
选择每个GROUP BY组中的第一行？ 1205
如何旋转Android模拟器显示？ 1031