Sir*_*mes 4 python winapi pywin32 python-2.7
早上好,
我在用 Python 创建的程序中遇到了一个特殊问题。看来,当我将文件从一个位置拖放到另一个位置时,并非所有文件都被模块注册为事件。
我一直在使用 win32file 和 win32con 来尝试获取与将文件从一个位置移动到另一个位置以进行处理相关的所有事件。
这是我的检测代码的一个片段:
import win32file
import win32con
def main():
path_to_watch = 'D:\\'
_file_list_dir = 1
# Create a watcher handle
_h_dir = win32file.CreateFile(
path_to_watch,
_file_list_dir,
win32con.FILE_SHARE_READ |
win32con.FILE_SHARE_WRITE |
win32con.FILE_SHARE_DELETE,
None,
win32con.OPEN_EXISTING,
win32con.FILE_FLAG_BACKUP_SEMANTICS,
None
)
while 1:
results = win32file.ReadDirectoryChangesW(
_h_dir,
1024,
True,
win32con.FILE_NOTIFY_CHANGE_FILE_NAME |
win32con.FILE_NOTIFY_CHANGE_DIR_NAME |
win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |
win32con.FILE_NOTIFY_CHANGE_SIZE |
win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |
win32con.FILE_NOTIFY_CHANGE_SECURITY,
None,
None
)
for _action, _file in results:
if _action == 1:
print 'found!'
if _action == 2:
print 'deleted!'
Run Code Online (Sandbox Code Playgroud)
我拖放了 7 个文件,但只找到了 4 个。
# found!
# found!
# found!
# found!
Run Code Online (Sandbox Code Playgroud)
我可以做什么来检测所有删除的文件?
[ActiveState.Docs]: win32file.ReadDirectoryChangesW(这是我可以在 [GitHub] 找到的最佳文档:mhammond/pywin32 - Python for Windows (pywin32) Extensions )是[MS.Docs]: ReadDirectoryChangesW 函数的包装器。这是它的说明(关于缓冲区):
\n\n\n当您第一次调用ReadDirectoryChangesW时,系统会分配一个缓冲区来存储更改信息。该缓冲区与目录句柄相关联,直到它被关闭,并且其大小在其生命周期内不会改变。调用此函数之间发生的目录更改将添加到缓冲区中,然后在下一次调用时返回。如果缓冲区溢出,则缓冲区的全部内容将被丢弃,lpBytesReturned参数包含零,并且ReadDirectoryChangesW函数将失败并显示错误代码ERROR_NOTIFY_ENUM_DIR。
\n
我的理解是,这是一个与作为参数传递的缓冲区不同的缓冲区( lpBuffer)
\n前者被传递给ReadDirectoryChangesW的每次调用(可能是不同的缓冲区(具有不同的大小)))
\n后者由系统分配,而前者在函数调用之前明确分配(由用户),并且
在函数调用之间存储数据(可能以某种原始格式),并且当调用函数时,缓冲区内容被复制(并格式化)到lpBuffer(如果同时没有溢出(并丢弃))
\n\n同步成功完成后,lpBuffer参数是一个格式化的缓冲区,写入缓冲区的字节数可在lpBytesReturned中找到中找到。如果传输的字节数为零,则缓冲区太大,系统无法分配,或者太小,无法提供目录或子树中发生的所有更改的详细信息。在这种情况下,您应该通过枚举目录或子树来计算更改。
\n
这在一定程度上证实了我之前的假设
\n无论如何,我拿了你的代码并对其进行了“一点”更改。
\n代码00.py:
\nimport sys\nimport msvcrt\nimport pywintypes\nimport win32file\nimport win32con\nimport win32api\nimport win32event\n\n\nFILE_LIST_DIRECTORY = 0x0001\nFILE_ACTION_ADDED = 0x00000001\nFILE_ACTION_REMOVED = 0x00000002\n\nASYNC_TIMEOUT = 5000\n\nBUF_SIZE = 65536\n\n\ndef get_dir_handle(dir_name, asynch):\n flags_and_attributes = win32con.FILE_FLAG_BACKUP_SEMANTICS\n if asynch:\n flags_and_attributes |= win32con.FILE_FLAG_OVERLAPPED\n dir_handle = win32file.CreateFile(\n dir_name,\n FILE_LIST_DIRECTORY,\n (win32con.FILE_SHARE_READ |\n win32con.FILE_SHARE_WRITE |\n win32con.FILE_SHARE_DELETE),\n None,\n win32con.OPEN_EXISTING,\n flags_and_attributes,\n None\n )\n return dir_handle\n\n\ndef read_dir_changes(dir_handle, size_or_buf, overlapped):\n return win32file.ReadDirectoryChangesW(\n dir_handle,\n size_or_buf,\n True,\n (win32con.FILE_NOTIFY_CHANGE_FILE_NAME |\n win32con.FILE_NOTIFY_CHANGE_DIR_NAME |\n win32con.FILE_NOTIFY_CHANGE_ATTRIBUTES |\n win32con.FILE_NOTIFY_CHANGE_SIZE |\n win32con.FILE_NOTIFY_CHANGE_LAST_WRITE |\n win32con.FILE_NOTIFY_CHANGE_SECURITY),\n overlapped,\n None\n )\n\n\ndef handle_results(results):\n for item in results:\n print(" {} {:d}".format(item, len(item[1])))\n _action, _ = item\n if _action == FILE_ACTION_ADDED:\n print(" found!")\n if _action == FILE_ACTION_REMOVED:\n print(" deleted!")\n\n\ndef esc_pressed():\n return msvcrt.kbhit() and ord(msvcrt.getch()) == 27\n\n\ndef monitor_dir_sync(dir_handle):\n idx = 0\n while True:\n print("Index: {:d}".format(idx))\n idx += 1\n results = read_dir_changes(dir_handle, BUF_SIZE, None)\n handle_results(results)\n if esc_pressed():\n break\n\n\ndef monitor_dir_async(dir_handle):\n idx = 0\n buffer = win32file.AllocateReadBuffer(BUF_SIZE)\n overlapped = pywintypes.OVERLAPPED()\n overlapped.hEvent = win32event.CreateEvent(None, False, 0, None)\n while True:\n print("Index: {:d}".format(idx))\n idx += 1\n read_dir_changes(dir_handle, buffer, overlapped)\n rc = win32event.WaitForSingleObject(overlapped.hEvent, ASYNC_TIMEOUT)\n if rc == win32event.WAIT_OBJECT_0:\n bufer_size = win32file.GetOverlappedResult(dir_handle, overlapped, True)\n results = win32file.FILE_NOTIFY_INFORMATION(buffer, bufer_size)\n handle_results(results)\n elif rc == win32event.WAIT_TIMEOUT:\n #print(" timeout...")\n pass\n else:\n print("Received {:d}. Exiting".format(rc))\n break\n if esc_pressed():\n break\n win32api.CloseHandle(overlapped.hEvent)\n\n\ndef monitor_dir(dir_name, asynch=False):\n dir_handle = get_dir_handle(dir_name, asynch)\n if asynch:\n monitor_dir_async(dir_handle)\n else:\n monitor_dir_sync(dir_handle)\n win32api.CloseHandle(dir_handle)\n\n\ndef main():\n print("Python {:s} on {:s}\\n".format(sys.version, sys.platform))\n asynch = True\n print("Attempting {}ynchronous mode using a buffer {:d} bytes long...".format("As" if async else "S", BUF_SIZE))\n monitor_dir(".\\\\test", asynch=asynch)\n\n\nif __name__ == "__main__":\n main()\nRun Code Online (Sandbox Code Playgroud)\n笔记:
\n输出:
\n\n\n\nRun Code Online (Sandbox Code Playgroud)\ne:\\Work\\Dev\\StackOverflow\\q049799109>dir /b test\n0123456789.txt\n01234567890123456789.txt\n012345678901234567890123456789.txt\n0123456789012345678901234567890123456789.txt\n01234567890123456789012345678901234567890123456789.txt\n012345678901234567890123456789012345678901234567890123456789.txt\n0123456789012345678901234567890123456789012345678901234567890123456789.txt\n01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\n012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\n0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\n\ne:\\Work\\Dev\\StackOverflow\\q049799109>\ne:\\Work\\Dev\\StackOverflow\\q049799109>"C:\\Install\\x64\\HPE\\OPSWpython\\2.7.10__00\\python.exe" code00.py\nPython 2.7.10 (default, Mar 8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32\n\nAttempting Synchronous mode using a buffer 512 bytes long...\nIndex: 0\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n deleted!\nIndex: 1\n (2, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n deleted!\nIndex: 2\n (2, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n deleted!\nIndex: 3\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n deleted!\n (2, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n deleted!\nIndex: 4\n (2, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n deleted!\nIndex: 5\n (2, u\'0123456789012345678901234567890123456789.txt\') 44\n deleted!\n (2, u\'012345678901234567890123456789.txt\') 34\n deleted!\nIndex: 6\n (2, u\'01234567890123456789.txt\') 24\n deleted!\n (2, u\'0123456789.txt\') 14\n deleted!\nIndex: 7\n (1, u\'0123456789.txt\') 14\n found!\nIndex: 8\n (3, u\'0123456789.txt\') 14\nIndex: 9\n (1, u\'01234567890123456789.txt\') 24\n found!\nIndex: 10\n (3, u\'01234567890123456789.txt\') 24\n (1, u\'012345678901234567890123456789.txt\') 34\n found!\n (3, u\'012345678901234567890123456789.txt\') 34\n (1, u\'0123456789012345678901234567890123456789.txt\') 44\n found!\nIndex: 11\n (3, u\'0123456789012345678901234567890123456789.txt\') 44\n (1, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n found!\n (3, u\'01234567890123456789012345678901234567890123456789.txt\') 54\nIndex: 12\nIndex: 13\n (1, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n found!\nIndex: 14\nIndex: 15\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n found!\nIndex: 16\n (3, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\nIndex: 17\n (1, u\'a\') 1\n found!\nIndex: 18\n (3, u\'a\') 1\n\ne:\\Work\\Dev\\StackOverflow\\q049799109>\ne:\\Work\\Dev\\StackOverflow\\q049799109>"C:\\Install\\x64\\HPE\\OPSWpython\\2.7.10__00\\python.exe" code00.py\nPython 2.7.10 (default, Mar 8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32\n\nAttempting Synchronous mode using a buffer 65536 bytes long...\nIndex: 0\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n deleted!\nIndex: 1\n (2, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n deleted!\nIndex: 2\n (2, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n deleted!\nIndex: 3\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n deleted!\nIndex: 4\n (2, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n deleted!\nIndex: 5\n (2, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n deleted!\nIndex: 6\n (2, u\'0123456789012345678901234567890123456789.txt\') 44\n deleted!\nIndex: 7\n (2, u\'012345678901234567890123456789.txt\') 34\n deleted!\n (2, u\'01234567890123456789.txt\') 24\n deleted!\n (2, u\'0123456789.txt\') 14\n deleted!\nIndex: 8\n (1, u\'0123456789.txt\') 14\n found!\nIndex: 9\n (3, u\'0123456789.txt\') 14\nIndex: 10\n (1, u\'01234567890123456789.txt\') 24\n found!\nIndex: 11\n (3, u\'01234567890123456789.txt\') 24\nIndex: 12\n (1, u\'012345678901234567890123456789.txt\') 34\n found!\nIndex: 13\n (3, u\'012345678901234567890123456789.txt\') 34\nIndex: 14\n (1, u\'0123456789012345678901234567890123456789.txt\') 44\n found!\nIndex: 15\n (3, u\'0123456789012345678901234567890123456789.txt\') 44\nIndex: 16\n (1, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n found!\n (3, u\'01234567890123456789012345678901234567890123456789.txt\') 54\nIndex: 17\n (1, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n found!\n (3, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n found!\nIndex: 18\n (3, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n (1, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n found!\n (3, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n (1, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n found!\n (3, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n found!\n (3, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\nIndex: 20\n (2, u\'a\') 1\n deleted!\n\ne:\\Work\\Dev\\StackOverflow\\q049799109>\ne:\\Work\\Dev\\StackOverflow\\q049799109>"C:\\Install\\x64\\HPE\\OPSWpython\\2.7.10__00\\python.exe" code00.py\nPython 2.7.10 (default, Mar 8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32\n\nAttempting Asynchronous mode using a buffer 512 bytes long...\nIndex: 0\nIndex: 1\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n deleted!\nIndex: 2\n (2, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n deleted!\nIndex: 3\n (2, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n deleted!\nIndex: 4\n (2, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n deleted!\nIndex: 5\n (2, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n deleted!\nIndex: 6\n (2, u\'0123456789012345678901234567890123456789.txt\') 44\n deleted!\nIndex: 7\n (2, u\'012345678901234567890123456789.txt\') 34\n deleted!\nIndex: 8\n (2, u\'01234567890123456789.txt\') 24\n deleted!\nIndex: 9\n (2, u\'0123456789.txt\') 14\n deleted!\nIndex: 10\nIndex: 11\nIndex: 12\n (1, u\'0123456789.txt\') 14\n found!\nIndex: 13\n (1, u\'01234567890123456789.txt\') 24\n found!\nIndex: 14\n (1, u\'012345678901234567890123456789.txt\') 34\n found!\nIndex: 15\n (3, u\'012345678901234567890123456789.txt\') 34\nIndex: 16\n (1, u\'0123456789012345678901234567890123456789.txt\') 44\n found!\n (3, u\'0123456789012345678901234567890123456789.txt\') 44\nIndex: 17\nIndex: 18\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n found!\nIndex: 19\nIndex: 20\n (1, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n found!\nIndex: 21\nIndex: 22\nIndex: 23\nIndex: 24\n\ne:\\Work\\Dev\\StackOverflow\\q049799109>\ne:\\Work\\Dev\\StackOverflow\\q049799109>"C:\\Install\\x64\\HPE\\OPSWpython\\2.7.10__00\\python.exe" code00.py\nPython 2.7.10 (default, Mar 8 2016, 15:02:46) [MSC v.1600 64 bit (AMD64)] on win32\n\nAttempting Asynchronous mode using a buffer 65536 bytes long...\nIndex: 0\nIndex: 1\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n deleted!\nIndex: 2\n (2, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n deleted!\nIndex: 3\n (2, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n deleted!\nIndex: 4\n (2, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n deleted!\nIndex: 5\n (2, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n deleted!\nIndex: 6\n (2, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n deleted!\nIndex: 7\n (2, u\'0123456789012345678901234567890123456789.txt\') 44\n deleted!\nIndex: 8\n (2, u\'012345678901234567890123456789.txt\') 34\n deleted!\n (2, u\'01234567890123456789.txt\') 24\n deleted!\nIndex: 9\n (2, u\'0123456789.txt\') 14\n deleted!\nIndex: 10\nIndex: 11\nIndex: 12\n (1, u\'0123456789.txt\') 14\n found!\nIndex: 13\n (1, u\'01234567890123456789.txt\') 24\n found!\nIndex: 14\n (1, u\'012345678901234567890123456789.txt\') 34\n found!\nIndex: 15\n (3, u\'012345678901234567890123456789.txt\') 34\n (1, u\'0123456789012345678901234567890123456789.txt\') 44\n found!\n (3, u\'0123456789012345678901234567890123456789.txt\') 44\nIndex: 16\n (1, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n found!\n (3, u\'01234567890123456789012345678901234567890123456789.txt\') 54\n (1, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n found!\n (3, u\'012345678901234567890123456789012345678901234567890123456789.txt\') 64\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n found!\nIndex: 17\n (3, u\'0123456789012345678901234567890123456789012345678901234567890123456789.txt\') 74\n (1, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n found!\n (3, u\'01234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 84\n (1, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n found!\n (3, u\'012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 94\n (1, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\n found!\n (3, u\'0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789.txt\') 104\nIndex: 18\nIndex: 19\n
评论:
\n底线是,没有任何方法可以避免丢失事件。采取的每一项措施都可以通过增加生成事件的数量来“击败”。
\n尽量减少损失:
\n缓冲区大小。这是您案例中的(主要)问题。不幸的是,文档非常清楚,没有关于它应该有多大的指导方针。浏览C论坛时我注意到64K是一个常见值。然而:
\n不可能有一个巨大的缓冲区,并且在失败的情况下减小其大小直到成功,因为这意味着丢失在计算缓冲区大小时生成的所有事件
\n即使64k足以容纳(多次)我在测试中生成的所有事件,有些事件仍然丢失。也许这是因为我一开始谈到的“神奇”缓冲区
\n尽可能减少事件的数量。就您而言,我注意到您只对添加和删除事件感兴趣(FILE_ACTION_ADDED和FILE_ACTION_REMOVED)。仅向ReadDirectoryChangesW指定适当的FILE_NOTIFY_CHANGE_*标志(例如,您不关心FILE_ACTION_MODIFIED,但在添加文件时收到它)
\n尝试将目录内容拆分为多个子目录并同时监视它们。例如,如果您只关心一个目录及其一堆子目录中发生的更改,则递归监视整个树是没有意义的,因为它很可能会产生大量无用的事件。无论如何,如果并行执行操作,请不要使用线程,因为GIL!([Python.Wiki]:GlobalInterpreterLock)。使用[Python.Docs]:多处理 - 基于进程的 \xe2\x80\x9cthreading\xe2\x80\x9d接口
\n提高循环中运行的代码的速度,以便在ReadDirectoryChangesW之外花费尽可能少的时间(当生成的事件可能溢出缓冲区时)。当然,下面的一些项目可能影响不大并且(也有一些不好的副作用),但我还是列出它们:
\n尽可能少地进行处理并尝试延迟它。也许在另一个进程中执行(因为GIL)
\n摆脱所有印刷品语句
\n而不是在脚本开头使用win32con.FILE_NOTIFY_CHANGE_FILE_NAME from win32con import FILE_NOTIFY_CHANGE_FILE_NAME ,并且仅使用FILE_NOTIFY_CHANGE_FILE_NAME(以避免在模块中查找变量)
\n不要使用函数(因为call / ret指令) - 对此不确定
\n尝试使用win32file.GetQueuedCompletionStatus方法来获取结果(仅限异步)
\n随着时间的推移,事情往往会变得更好(当然也有例外),尝试切换到更新的Python版本。也许它会跑得更快
\n使用C - 这可能是不可取的,但它可能有一些好处:
\nPyWin32不会在Python和C之间进行来回转换 - 但我没有使用探查器来检查它们花费了多少时间
\nlpCompletionRoutine(PyWin32不提供)也将可用,也许它更快
\n作为替代方案,可以使用CTypes调用C,但这需要一些工作,而且我觉得不值得
\n| 归档时间: |
|
| 查看次数: |
3811 次 |
| 最近记录: |