Having multiple simultaneous writers (no reader) to a single file. Is it possible to accomplish in a performant way in .NET?

Question

I'm developing a multiple segment file downloader. To accomplish this task I'm currently creating as many temporary files on disk as segments I have (they are fixed in number during the file downloading). In the end I just create a new file f and copy all the segments' contents onto f.

I was wondering if there's not a better way to accomplish this. My idealization is of initially creating f in its full-size and then have the different threads write directly onto their portion. There need not to be any kind of interaction between them. We can assume any of them will start at its own starting point in the file and then only fill information sequentially in the file until its task is over.

I've heard about Memory-Mapped files (http://msdn.microsoft.com/en-us/library/dd997372(v=vs.110).aspx) and I'm wondering if they are the solution to my problem or not.

Thanks

Answer 1

使用内存映射 API 是绝对可行的，并且它可能会表现得很好 - 因为建议进行一些测试。

如果您想寻找可能的替代实现，我有以下建议。

• 创建一个静态堆栈数据结构，下载线程可以在下载后立即推送每个文件段。

• 让一个单独的线程侦听堆栈上的推送通知。以单线程方式弹出堆栈文件段并将每个段保存到目标文件中。

通过遵循上述模式，您通过在两者之间放置一个堆栈容器，将文件段的下载和保存到常规文件中分开。

根据堆栈处理的实现，您将能够用很少的线程锁定来实现这一点，这将最大限度地提高性能。

这样做的优点是您可以 100% 控制正在发生的事情，并且解决方案可能更便携（如果这应该是一个问题的话）。

您所做的堆栈解耦模式也可以非常通用地实现，甚至可能在将来被重用。

这个的实现并不那么复杂，可能与内存映射 api 周围需要完成的实现相当。

玩得开心...

/安德斯