9 .net compression crc32 gzip error-detection
前几天我遇到了问题GZipStream没有检测到损坏的数据(甚至CRC32通过)？(其中很可能是一个"重复",我对这个主题有不同的感受.我也是那个将CRC32添加到标题中的人,但回想起来,与帖子的其余部分感觉不合适).在我自己探讨了这个问题后,我认为这个问题比最初描述的其他问题要大得多.
我扩展了另一个问题,并使测试代码在LINQPad下可运行,并试图更好地展示CRC32(循环冗余校验)问题,如果确实存在的话.(由于代码只是基于原始版本的轻微修改,因此测试设置/方法可能存在缺陷,或者存在另一个奇怪的怪癖/ PEBCAK.)
The results are odd because the corrupt data is not always causing an (any!) Exception to be raised. Note that only sometimes does the CRC32 check seem to actually be "working". The corrupt bytes that cause the index-out-of-range/bad header/bad footer can be ignored because we can assume these are killing the decompression prior to the CRC32 check (which is perfectly understandable, even if the IndexOutOfRangeException should likely be wrapped by a InvalidDataException) so,
Why is the CRC32 check significantly less reliable than it ought to be? (Why is it the case that there is "Invalid data (No Exception)" below?)
Since the GZip footer contains both the CRC32 and length of the uncompressed data is seems that the error detection rate should be "significantly higher" -- that is, I would not expect a single failing case below, much less a number of undetected corrupted streams. (Of course it's nice to detect a corrupt steam ASAP: but the final safe-guard checksum seems to be downright ignored in cases.)
Format is CorruptByteIndex+FailedDetections: Message:
0+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. 1+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream. 2+0: System.IO.InvalidDataException:The compression mode specified in GZip header is unknown. 3+0: Good data (No Exception) 4+0: Good data (No Exception) 5+0: Good data (No Exception) 6+0: Good data (No Exception) 7+0: Good data (No Exception) 8+0: Good data (No Exception) 9+0: Good data (No Exception) 10+0: System.IO.InvalidDataException:Unknown block type. Stream might be corrupted. 11+1: Invalid data (No Exception) 12+1: System.IO.InvalidDataException:Found invalid data while decoding. 13+1: System.IO.InvalidDataException:Found invalid data while decoding. 14+1: System.IO.InvalidDataException:Found invalid data while decoding. 15+1: System.IO.InvalidDataException:Found invalid data while decoding. 16+1: System.IO.InvalidDataException:Found invalid data while decoding. 17+2: Invalid data (No Exception) 18+2: System.IO.InvalidDataException:Found invalid data while decoding. 19+2: System.IndexOutOfRangeException:Index was outside the bounds of the array. 20+2: System.IndexOutOfRangeException:Index was outside the bounds of the array. 21+3: Invalid data (No Exception) 22+3: System.IndexOutOfRangeException:Index was outside the bounds of the array. 23+3: System.IndexOutOfRangeException:Index was outside the bounds of the array. 24+4: Invalid data (No Exception) 25+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 26+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 27+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 28+4: System.IndexOutOfRangeException:Index was outside the bounds of the array. 29+5: Invalid data (No Exception) 30+5: System.IndexOutOfRangeException:Index was outside the bounds of the array. 31+6: Invalid data (No Exception) 32+7: Invalid data (No Exception) 33+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 34+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 35+7: System.IndexOutOfRangeException:Index was outside the bounds of the array. 36+8: Invalid data (No Exception) 37+8: System.IndexOutOfRangeException:Index was outside the bounds of the array. 38+8: System.IndexOutOfRangeException:Index was outside the bounds of the array. 39+9: Invalid data (No Exception) 40+9: System.IndexOutOfRangeException:Index was outside the bounds of the array. 41+9: System.IndexOutOfRangeException:Index was outside the bounds of the array. 42+10: Invalid data (No Exception) 43+10: System.IO.InvalidDataException:Found invalid data while decoding. 44+10: System.IndexOutOfRangeException:Index was outside the bounds of the array. 45+10: System.IO.InvalidDataException:Found invalid data while decoding. 46+11: Invalid data (No Exception) 47+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 48+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 49+11: System.IndexOutOfRangeException:Index was outside the bounds of the array. 50+12: Invalid data (No Exception) 51+12: System.IndexOutOfRangeException:Index was outside the bounds of the array. 52+12: System.IndexOutOfRangeException:Index was outside the bounds of the array. 53+13: Invalid data (No Exception) 54+13: System.IndexOutOfRangeException:Index was outside the bounds of the array. 55+14: Invalid data (No Exception) 56+14: System.IndexOutOfRangeException:Index was outside the bounds of the array. 57+15: Invalid data (No Exception) 58+15: System.IndexOutOfRangeException:Index was outside the bounds of the array. 59+15: System.IndexOutOfRangeException:Index was outside the bounds of the array. 60+16: Invalid data (No Exception) 61+17: Invalid data (No Exception) 62+18: Invalid data (No Exception) 63+19: Invalid data (No Exception) 64+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 65+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 66+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 67+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 68+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 69+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 70+19: System.IO.InvalidDataException:Found invalid data while decoding. 71+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 72+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 73+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 74+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 75+19: System.IO.InvalidDataException:Found invalid data while decoding. 76+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 77+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 78+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 79+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 80+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 81+19: System.IO.InvalidDataException:Found invalid data while decoding. 82+19: System.IndexOutOfRangeException:Index was outside the bounds of the array. 83+20: Invalid data (No Exception) 84+21: Invalid data (No Exception) 85+22: Invalid data (No Exception) 86+22: System.IndexOutOfRangeException:Index was outside the bounds of the array. 87+23: Invalid data (No Exception) 88+24: Invalid data (No Exception) 89+25: Invalid data (No Exception) 90+25: System.IndexOutOfRangeException:Index was outside the bounds of the array. 91+26: Invalid data (No Exception) 92+26: System.IO.InvalidDataException:Found invalid data while decoding. 93+26: System.IndexOutOfRangeException:Index was outside the bounds of the array. 94+27: Invalid data (No Exception) 95+27: System.IndexOutOfRangeException:Index was outside the bounds of the array. 96+27: System.IndexOutOfRangeException:Index was outside the bounds of the array. 97+28: Invalid data (No Exception) 98+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 99+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 100+29: Invalid data (No Exception) 101+30: Invalid data (No Exception) 102+31: Invalid data (No Exception) 103+32: Invalid data (No Exception) 104+32: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 105+33: Invalid data (No Exception) 106+34: Invalid data (No Exception) 107+35: Invalid data (No Exception) 108+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 109+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 110+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 111+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 112+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 113+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 114+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 115+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 116+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 117+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 118+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 119+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 120+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 121+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 122+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 123+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 124+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 125+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 126+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 127+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 128+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 129+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 130+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 131+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 132+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 133+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 134+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 135+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 136+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 137+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 138+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 139+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 140+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 141+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 142+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 143+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 144+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 145+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 146+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 147+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 148+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 149+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 150+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 151+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 152+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 153+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 154+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 155+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 156+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 157+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 158+36: Invalid data (No Exception) 159+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 160+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 161+37: Invalid data (No Exception) 162+38: Invalid data (No Exception) 163+39: Invalid data (No Exception) 164+40: Invalid data (No Exception) 165+41: Invalid data (No Exception) 166+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 167+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 168+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 169+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data. 170+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 171+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 172+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size. 173+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.
Here is the test which is copy'n'paste runnable in LINQPad (for .NET 3.5 and 4, use "as C# statements" mode):
string sample = "This is a compression test of microsoft .net gzip compression method and decompression methods";
var encoding = new ASCIIEncoding();
var data = encoding.GetBytes(sample);
string sampleOut = null;
byte[] cmpData;
// Compress
using (var cmpStream = new MemoryStream())
{
using (var hgs = new System.IO.Compression.GZipStream(cmpStream, System.IO.Compression.CompressionMode.Compress))
{
hgs.Write(data, 0, data.Length);
}
cmpData = cmpStream.ToArray();
}
int corruptBytesNotDetected = 0;
// corrupt data byte by byte
for (var byteToCorrupt = 0; byteToCorrupt < cmpData.Length; byteToCorrupt++)
{
var corruptData = new List<byte>(cmpData).ToArray();
// corrupt the data
corruptData[byteToCorrupt]++;
using (var decomStream = new MemoryStream(corruptData))
{
using (var hgs = new System.IO.Compression.GZipStream(decomStream, System.IO.Compression.CompressionMode.Decompress))
{
using (var reader = new StreamReader(hgs))
{
string message;
try
{
sampleOut = reader.ReadToEnd();
// if we get here, the corrupt data was not detected by GZipStream
// ... okay so long as the correct data is extracted
if (!sample.SequenceEqual(sampleOut)) {
corruptBytesNotDetected++;
message = "Invalid data (No Exception)";
} else {
message = "Good data (No Exception)";
}
}
catch(Exception ex)
{
message = (ex.GetType() + ":" + ex.Message);
}
string.Format("{0}+{1}: {2}",
byteToCorrupt, corruptBytesNotDetected, message).Dump();
}
}
}
}
以下是.NET 3.5中的压缩数据(GZipStream在"压缩"小有效负载方面非常糟糕,但它是"无法解决"问题,因为流在技术上仍然有效):
1F 8B 08 00 00 00 00 00 04 00 ED BD 07 60 1C 49 96 25 26 2F 6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10 EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D 66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B 9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80 AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D 3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE F2 FE 17 CD FF 03 5C 51 5E 27 5E 00 00 00
(而且,就像giggles一样,在.NET 4中它会生成一个稍大/不同的压缩流.)
1F 8B 08 00 00 00 00 00 04 00 EC BD 07 60 1C 49 96 25 26 2F 6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10 EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D 66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B 9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80 AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A AC EA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D 3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CE F2 FE 17 CD FF 13 00 00 FF FF 5C 51 5E 27 5E 00 00 00
补充说明:
在这种情况下,测试可能会有微妙的缺陷.当GZipStream"无法检测到腐败"(无异常),然后从StreamReader的读出的数据是""(一个空字符串):在这种情况下,为什么ReadToEnd() 没有引发异常(IOException异常或其他)？
因此它不是 GZipStream而是StreamReader在这里"古怪"或者这仍然是GZipStream的问题(不抛出异常)？是否有一些正确的方法来可靠地处理这个用例？(考虑来自当前位置的输入流确实为空.)
Mar*_*ler 11
.NET [4和以前]用户在任何情况下都不应使用Microsoft提供的GZipStream或DeflateStream类,除非Microsoft完全用有效的东西替换它们.请改用DotNetZip库.
.NET Framework 4.5及更高版本修复了压缩问题,GZipStream和DeflateStream在这些版本中使用zlib.我不知道下面引用的CRC问题是否已修复.
CRC错误不仅没有修复,而且微软决定不修复它!
我的回答为什么我的C#gzip会产生比Fiddler或PHP更大的文件？表明此行为未反映gzip损坏检测的正确实现.在所有测试的案例中,错误将通过适当的实现来检测.(该回复还指出为什么有七个案例会产生良好的数据.)
另一个问题是:这种"压缩"方法如何从94字节的字符串产生174字节的输出？特别是看看字符串是如何可压缩的 - gzip将其减少到84字节,即使是18字节的头文件和预告片的开销.你能提供174字节的十六进制转储吗？
| 归档时间： |
|
| 查看次数： |
5217 次 |
| 最近记录： |