FileStream.WriteFile的奇怪行为

Question

我正在研究一个在大文件上进行大量读/写随机访问的程序(直到64 GB).文件是专门构建的,为了对它们进行访问,我创建了一个框架; 过了一会儿,我试着测试它的性能,我注意到在预分配的文件顺序写入操作太慢而不能接受.经过多次测试后,我在没有框架的情况下复制了行为(只有FileStream方法); 这是(使用我的硬件)复制问题的代码部分:

FileStream fs = new FileStream("test1.vhd", FileMode.Open);
byte[] buffer = new byte[256 * 1024];
Random rand = new Random();
rand.NextBytes(buffer);
DateTime start, end;
double ellapsed = 0.0;
long startPos, endPos;

BinaryReader br = new BinaryReader(fs);
br.ReadUInt32();
br.ReadUInt32();
for (int i = 0; i < 65536; i++)
    br.ReadUInt16();

br = null;

startPos = 0;   // 0
endPos = 4294967296;    // 4GB
for (long index = startPos; index < endPos; index += buffer.Length)
{
    start = DateTime.Now;
    fs.Write(buffer, 0, buffer.Length);
    end = DateTime.Now;
    ellapsed += (end - start).TotalMilliseconds;
}

不幸的是,这个问题似乎是不可预测的,所以有时它"有效",有时则不然.但是,使用Process Monitor我发现了以下事件:

Operation   Result  Detail
WriteFile   SUCCESS Offset: 1.905.655.816, Length: 262.144
WriteFile   SUCCESS Offset: 1.905.917.960, Length: 262.144
WriteFile   SUCCESS Offset: 1.906.180.104, Length: 262.144
WriteFile   SUCCESS Offset: 1.906.442.248, Length: 262.144
WriteFile   SUCCESS Offset: 1.906.704.392, Length: 262.144
WriteFile   SUCCESS Offset: 1.906.966.536, Length: 262.144
ReadFile    SUCCESS Offset: 1.907.228.672, Length: 32.768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile   SUCCESS Offset: 1.907.228.680, Length: 262.144
ReadFile    SUCCESS Offset: 1.907.355.648, Length: 32.768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile    SUCCESS Offset: 1.907.490.816, Length: 32.768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile   SUCCESS Offset: 1.907.490.824, Length: 262.144
ReadFile    SUCCESS Offset: 1.907.617.792, Length: 32.768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
ReadFile    SUCCESS Offset: 1.907.752.960, Length: 32.768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
WriteFile   SUCCESS Offset: 1.907.752.968, Length: 262.144

也就是说,在重写了大约2 GB 之后,每次都FileStream.Write开始调用,这个问题一直持续到过程结束; 此外,问题开始的偏差似乎是随机的.我已经在方法中逐步调试,并且我已经验证了实际上是内部调用的(Win32 API).ReadFileWriteFileFileStream.WriteWriteFileReadFile

最后一点; 我不认为这是一个文件碎片问题:我用contig对我的文件进行了碎片整理!

Answer 1

我相信这与 FileStream.Write / Read 和 2GB 限制有关。您是否在 32 位进程中运行此程序？我找不到任何关于此的具体文档，但这里有一个听起来相同的MSDN 论坛问题。您可以尝试在 64 位进程中运行它。

不过，我同意使用内存映射文件可能是更好的方法。