避免在目录中一次更改错误太多

Question

如何避免C#中FileSystemWatcher的错误？

目录中一次更改太多

我必须检测网络共享上的所有更改.InternalBufferSize增加到8192*128

Answer 1

你应该做两件事:

1. 设置InternalBufferSize为支持的最大值(65536).您将其设置为"8192*128"的尝试大于文档中列出的最大支持值,因此您可能根本没有增加缓冲区大小.
2. 将事件从FileSystemWatcher后台线程排队以进行处理.

这是第二点,不太了解,真的应该在MSDN上记录.在内部,FileSystemWatcher将更改事件排队到您设置上面大小的内部缓冲区.但重要的是,只有在事件处理程序返回后才会从该缓冲区中删除项目.这意味着事件处理程序引入的每个开销周期都会增加缓冲区填满的可能性.您应该做的是尽快清除有限的队列FileSystemWatcher,并将事件移动到您自己的无限队列中,以您可以处理的速率处理,或者如果您愿意,可以丢弃,但是有一些智能.

这基本上就是我在代码中所做的.首先,我开始自己的调度程序线程:

Dispatcher changeDispatcher = null;
ManualResetEvent changeDispatcherStarted = new ManualResetEvent(false);
Action changeThreadHandler = () =>
{
    changeDispatcher = Dispatcher.CurrentDispatcher;
    changeDispatcherStarted.Set();
    Dispatcher.Run();
};
new Thread(() => changeThreadHandler()) { IsBackground = true }.Start();
changeDispatcherStarted.WaitOne();

然后我创建了观察者.请注意设置的缓冲区大小.就我而言,我只关注目标目录中的更改,而不是子目录:

FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.InternalBufferSize = 64 * 1024;
watcher.IncludeSubdirectories = false;

现在我附加了我的事件处理程序,但是在这里我将它们调用到我的调度程序而不是在观察程序线程中同步运行它们.是的,调度员将按顺序处理事件:

watcher.Changed += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnChanged(sender, e)));
watcher.Created += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnCreated(sender, e)));
watcher.Deleted += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnDeleted(sender, e)));
watcher.Renamed += (sender, e) => changeDispatcher.BeginInvoke(new Action(() => OnRenamed(sender, e)));

最后,在处理FileSystemWatcher掉(你正在这样做,对吗？)之后,你需要关闭你的调度员:

watcher.Dispose()
changeDispatcher.BeginInvokeShutdown(DispatcherPriority.Normal);

就是这样.在网络和本地场景中,我自己都遇到了这个问题.使用这种方法后,我无法再次生成此错误,即使在尽可能快地将空文件敲打到监视目录时也是如此.如果你曾经设法以某种方式耗尽这种情况下的缓冲区(我不确定是否可能,API上游可能更慢),这里还有进一步优化的空间.只要您的调度程序超过了"临界点",发送方无法以比发送方式更快的速度发布事件,您将永远不会得到积压,因此永远不会破坏缓冲区.我相信这种方法可以让你进入安全区域.

Answer 2

我想我可能已经找到了一种可以帮助显着提高缓冲区使用率的模式。

此类的问题在于，在事件的委托完成运行之前，它无法释放用于保存该信息的内存。

在我的一生中，我不知道为什么将最大 InternalBufferSize 设置为 64Kb，但是有了这个想法，您将更有效地使用那个小缓冲区。

如果不是在委托内部执行操作，而是简单地将它们排队并推迟它们的执行以供后台工作人员使用，则它使用的内存量将小得多。

事实上，我不知道为什么这个类本身一开始就没有实现这样的东西。

下面的代码只是该想法的示例代码，不应在生产环境中使用，但它大大增加了我可以复制和跟踪的文件数量。

using System;
using System.Collections.Concurrent;
using System.ComponentModel;
using System.IO;
using System.Threading;
using NUnit.Framework;

namespace Soundnet.Synchronisation.FileSystemWatcherTests
{


  /// <summary>
  /// 
  /// </summary>
  [TestFixture]
  public class Tests
  {

    static readonly ConcurrentQueue<Change> ChangesQueue = new ConcurrentQueue<Change>();
    const string Destination = @"c:\Destination";
    const string Source = @"c:\Source";

    /// <summary>
    /// Tests this instance.
    /// </summary>
    [Test]
    public void Test()
    {
      var changesBackgroundWorker = new BackgroundWorker();
      changesBackgroundWorker.DoWork += ChangesBackgroundWorkerOnDoWork;
      changesBackgroundWorker.RunWorkerAsync();
      var fileSystemWatcher = new FileSystemWatcher
                                {
                                  Path = Source,
                                  EnableRaisingEvents = true,
                                  IncludeSubdirectories = true,
                                  NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.LastAccess | NotifyFilters.CreationTime | NotifyFilters.FileName | NotifyFilters.DirectoryName,
                                  InternalBufferSize = 65536
                                };
      fileSystemWatcher.Created += FileSystemWatcherOnCreated;
      fileSystemWatcher.Deleted += FileSystemWatcherOnDeleted;
      fileSystemWatcher.Renamed += FileSystemWatcherOnRenamed;
      fileSystemWatcher.Error += FileSystemWatcherOnError;

      while (true)
        Thread.Sleep(1000000);
    }

    /// <summary>
    /// Changeses the background worker configuration document work.
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="doWorkEventArgs">The <see cref="DoWorkEventArgs"/> instance containing the event data.</param>
    /// <exception cref="System.ArgumentOutOfRangeException"></exception>
    private static void ChangesBackgroundWorkerOnDoWork(object sender, DoWorkEventArgs doWorkEventArgs)
    {
      while (true)
      {
        Change change;
        if (ChangesQueue.TryDequeue(out change))
        {
          var backgroundWorker = new BackgroundWorker();
          switch (change.ChangeType)
          {
            case WatcherChangeTypes.Created:
              backgroundWorker.DoWork += (o, args) =>
                                          {
                                            var fileSystemType = GetFileSystemType(change.FullPath);

                                            var newItem = Path.Combine(Destination, change.Name);
                                            while (true)
                                            {
                                              try
                                              {
                                                switch (fileSystemType)
                                                {
                                                  case FileSystemType.File:
                                                    File.Copy(change.FullPath, newItem, true);
                                                    break;
                                                  case FileSystemType.Directory:
                                                    var directorySecurity =
                                                      Directory.GetAccessControl(change.FullPath);
                                                    Directory.CreateDirectory(newItem, directorySecurity);
                                                    break;
                                                  case FileSystemType.NotExistant:
                                                    break;
                                                }
                                                return;
                                              }
                                              catch (IOException exception)
                                              {
                                                Thread.Sleep(100);
                                                Console.WriteLine(exception.Message);
                                              }
                                            }
                                          };
              break;
            case WatcherChangeTypes.Deleted:
              backgroundWorker.DoWork += (o, args) =>
              {
                var itemToDelete = Path.Combine(Destination, change.Name);
                var fileSystemType = GetFileSystemType(itemToDelete);
                switch (fileSystemType)
                {
                  case FileSystemType.File:
                    File.Delete(itemToDelete);
                    break;
                  case FileSystemType.Directory:
                    Directory.Delete(itemToDelete, true);
                    break;
                }
              };
              break;
            case WatcherChangeTypes.Changed:
              backgroundWorker.DoWork += (o, args) =>
              {
                var fileSystemType = GetFileSystemType(change.FullPath);
                var newItem = Path.Combine(Destination, change.Name);
                switch (fileSystemType)
                {
                  case FileSystemType.File:
                    File.Copy(change.FullPath, newItem, true);
                    break;
                }
              };
              break;
            case WatcherChangeTypes.Renamed:
              backgroundWorker.DoWork += (o, args) =>
              {
                var fileSystemType = GetFileSystemType(change.FullPath);
                var oldItem = Path.Combine(Destination, change.OldName);
                var newItem = Path.Combine(Destination, change.Name);
                switch (fileSystemType)
                {
                  case FileSystemType.File:
                    if (File.Exists(oldItem))
                      File.Move(oldItem, newItem);
                    break;
                  case FileSystemType.Directory:
                    if (Directory.Exists(oldItem))
                      Directory.Move(oldItem, newItem);
                    break;
                }
              };
              break;
            case WatcherChangeTypes.All:
              break;
            default:
              throw new ArgumentOutOfRangeException();
          }
          backgroundWorker.RunWorkerAsync();
        }
      }
    }

    /// <summary>
    /// Files the system watcher configuration created.
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="fileSystemEventArgs">The <see cref="FileSystemEventArgs"/> instance containing the event data.</param>
    private static void FileSystemWatcherOnCreated(object sender, FileSystemEventArgs fileSystemEventArgs)
    {
      ChangesQueue.Enqueue(new Change
      {
        ChangeType = WatcherChangeTypes.Created,
        FullPath = fileSystemEventArgs.FullPath,
        Name = fileSystemEventArgs.Name
      });
    }

    /// <summary>
    /// Files the system watcher configuration deleted.
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="fileSystemEventArgs">The <see cref="FileSystemEventArgs"/> instance containing the event data.</param>
    private static void FileSystemWatcherOnDeleted(object sender, FileSystemEventArgs fileSystemEventArgs)
    {
      ChangesQueue.Enqueue(new Change
      {
        ChangeType = WatcherChangeTypes.Deleted,
        FullPath = fileSystemEventArgs.FullPath,
        Name = fileSystemEventArgs.Name
      });
    }

    /// <summary>
    /// Files the system watcher configuration error.
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="errorEventArgs">The <see cref="ErrorEventArgs"/> instance containing the event data.</param>
    private static void FileSystemWatcherOnError(object sender, ErrorEventArgs errorEventArgs)
    {
      var exception = errorEventArgs.GetException();
      Console.WriteLine(exception.Message);
    }

    /// <summary>
    /// Files the system watcher configuration renamed.
    /// </summary>
    /// <param name="sender">The sender.</param>
    /// <param name="fileSystemEventArgs">The <see cref="RenamedEventArgs"/> instance containing the event data.</param>
    private static void FileSystemWatcherOnRenamed(object sender, RenamedEventArgs fileSystemEventArgs)
    {
      ChangesQueue.Enqueue(new Change
      {
        ChangeType = WatcherChangeTypes.Renamed,
        FullPath = fileSystemEventArgs.FullPath,
        Name = fileSystemEventArgs.Name,
        OldFullPath = fileSystemEventArgs.OldFullPath,
        OldName = fileSystemEventArgs.OldName
      });
    }

    /// <summary>
    /// Gets the type of the file system.
    /// </summary>
    /// <param name="fullPath">The full path.</param>
    /// <returns></returns>
    private static FileSystemType GetFileSystemType(string fullPath)
    {
      if (Directory.Exists(fullPath))
        return FileSystemType.Directory;
      if (File.Exists(fullPath))
        return FileSystemType.File;
      return FileSystemType.NotExistant;
    }
  }

  /// <summary>
  /// Type of file system object
  /// </summary>
  internal enum FileSystemType
  {
    /// <summary>
    /// The file
    /// </summary>
    File,
    /// <summary>
    /// The directory
    /// </summary>
    Directory,
    /// <summary>
    /// The not existant
    /// </summary>
    NotExistant
  }

  /// <summary>
  /// Change information
  /// </summary>
  public class Change
  {
    /// <summary>
    /// Gets or sets the type of the change.
    /// </summary>
    /// <value>
    /// The type of the change.
    /// </value>
    public WatcherChangeTypes ChangeType { get; set; }

    /// <summary>
    /// Gets or sets the full path.
    /// </summary>
    /// <value>
    /// The full path.
    /// </value>
    public string FullPath { get; set; }

    /// <summary>
    /// Gets or sets the name.
    /// </summary>
    /// <value>
    /// The name.
    /// </value>
    public string Name { get; set; }

    /// <summary>
    /// Gets or sets the old full path.
    /// </summary>
    /// <value>
    /// The old full path.
    /// </value>
    public string OldFullPath { get; set; }

    /// <summary>
    /// Gets or sets the old name.
    /// </summary>
    /// <value>
    /// The old name.
    /// </value>
    public string OldName { get; set; }
  }
}

Answer 3

从MSDN;

Windows 操作系统会通知您的组件由FileSystemWatcher. 如果短时间内发生多次变化，缓冲区可能会溢出。这会导致组件失去对目录中更改的跟踪，并且它只会提供全面通知。使用该属性增加缓冲区大小的成本InternalBufferSize很高，因为它来自无法换出到磁盘的非分页内存，因此请保持缓冲区足够小但足够大，以免错过任何文件更改事件。为了避免缓冲区溢出，请使用NotifyFilter和 IncludeSubdirectories属性，以便您可以过滤掉不需要的更改通知。

Answer 4

我的经验是 FileSystemWatcher 并不总是最可靠的。您可以指定过滤器来缩小您正在观看的文件的范围（NotifyFilter），或增加缓冲区大小。

但根据您的要求，您可能还想采用另一种方式，例如每 x 秒轮询一次以获取文件列表。但随后您可能需要告诉我们更多有关您的业务案例的信息。