如何在大型稀疏文件上创建快速有效的文件流写入

Question

我有一个应用程序,可以在多个段中写入大文件.我使用FileStream.Seek来定位每个wirte.看来,当我在稀疏文件中的深位置调用FileStream.Write时,write会在所有前面的字节上触发"回填"操作(写入0),这很慢.

有没有更有效的方法来处理这种情况？

以下代码演示了该问题.初始写入在我的机器上大约需要370 MS.

    public void WriteToStream()
    {
        DateTime dt;
        using (FileStream fs = File.Create("C:\\testfile.file"))
        {   
            fs.SetLength(1024 * 1024 * 100);
            fs.Seek(-1, SeekOrigin.End);
            dt = DateTime.Now;
            fs.WriteByte(255);              
        }

        Console.WriteLine(@"WRITE MS: " + DateTime.Now.Subtract(dt).TotalMilliseconds.ToString());
    }

Answer 1

NTFS确实支持稀疏文件,但是在没有p /调用某些本机方法的情况下,无法在.net中执行此操作.

将文件标记为稀疏文件并不是很难,只要知道一旦文件被标记为稀疏文件,它就永远不能转换回非稀疏文件,除非将整个文件复制到新的非稀疏文件中.

用例示例

class Program
{
    [DllImport("Kernel32.dll", SetLastError = true, CharSet = CharSet.Auto)]
    private static extern bool DeviceIoControl(
        SafeFileHandle hDevice,
        int dwIoControlCode,
        IntPtr InBuffer,
        int nInBufferSize,
        IntPtr OutBuffer,
        int nOutBufferSize,
        ref int pBytesReturned,
        [In] ref NativeOverlapped lpOverlapped
    );

    static void MarkAsSparseFile(SafeFileHandle fileHandle)
    {
        int bytesReturned = 0;
        NativeOverlapped lpOverlapped = new NativeOverlapped();
        bool result =
            DeviceIoControl(
                fileHandle,
                590020, //FSCTL_SET_SPARSE,
                IntPtr.Zero,
                0,
                IntPtr.Zero,
                0,
                ref bytesReturned,
                ref lpOverlapped);
        if(result == false)
            throw new Win32Exception();
    }

    static void Main()
    {
        //Use stopwatch when benchmarking, not DateTime
        Stopwatch stopwatch = new Stopwatch();

        stopwatch.Start();
        using (FileStream fs = File.Create(@"e:\Test\test.dat"))
        {
            MarkAsSparseFile(fs.SafeFileHandle);

            fs.SetLength(1024 * 1024 * 100);
            fs.Seek(-1, SeekOrigin.End);
            fs.WriteByte(255);
        }
        stopwatch.Stop();

        //Returns 2 for sparse files and 1127 for non sparse
        Console.WriteLine(@"WRITE MS: " + stopwatch.ElapsedMilliseconds); 
    }
}

一旦文件被标记为稀疏文件,它现在的行为就像你排除了它在评论中的行为一样.您不需要编写一个字节来将文件标记为设置大小.

static void Main()
{
    string filename = @"e:\Test\test.dat";

    using (FileStream fs = new FileStream(filename, FileMode.Create))
    {
        MarkAsSparseFile(fs.SafeFileHandle);

        fs.SetLength(1024 * 1024 * 25);
    }
}