在 Linux 中将大量文件从一个目录复制到另一个目录

Question

我有一个包含大约 280,000 个文件的目录。我想将它们移动到另一个目录。

如果我使用cp或mv然后我得到一个错误“参数列表太长”。

如果我写一个脚本

for file in ls *; do
   cp {source} to {destination} 
done

然后，由于该ls命令，其性能下降。

我怎样才能做到这一点？

Answer 1

使用rsync：

$ rsync -a {source}/ {destination}/

例如

$ rsync -a /some/path/to/src/ /other/path/to/dest/

（注意尾随 /s）

注意：如果这是一个冗长的操作，并且您想在复制过程中看到一些进度指示，您可以添加-v（详细）选项，然后列出正在复制的每个文件，或者考虑使用该--progress选项，以获得更简洁的进度输出。

Answer 2

我在这里的答案中遗漏了两个双胞胎，所以我再添加一个。

虽然这让我想起了添加另一个标准答案......

这里有两个问题：

我有一个包含大约 280,000 个文件的目录。

大多数工具不能很好地扩展这个数量的文件。不仅仅是大多数 Linux 工具或 Windows 工具，还有相当多的程序。这可能包括您的文件系统。长期的解决方案是“好吧，那就不要那样做”。如果您有不同的文件，但它们在不同的目录中。如果不希望将来继续遇到问题。

话虽如此，让我们转向您的实际问题：

如果我使用 cp 或 mv 然后我得到一个错误“参数列表太长”

This is caused by expansion of * by the shell. The shell has limited space for the result and it runs out. This means any command with an * expanded by the shell will run into the same problem. You will either need to expand fewer options at the same time, or use a different command.

One alternate command used often when you run into this problem is find. There are already several answers showing how to use it, so I am not going to repeat all that. I am however going to point out the difference between \; and +, since this can make a huge performance difference and hook nicely into the previous expansion explanation.

find /path/to/search --name "*.txt" -exec command {} \;

Will find all files under path/to/search/ and exec a command with it, but notice the quotes around the *. That feeds the * to the command. If we did not encapsulate it or escape it then the shell would try to expand it and we would get the same error.

Lastly, I want to mention something about {}. These brackets get replaced by the content found by find. If you end the command with a semicolom ; (one which you need to escape from the shell, hence the \;'s in the examples) then the results are passed one by one. This means that you will execute 280000 mv commands. One for each file. This might be slow.

Alternatively you can end with +. This will pass as many arguments as possible at the same time. If bash can handle 2000 arguments, then find /path -name "*filetype" -exec some_move {}+ will call the some_move command about 140 times, each time with 2000 arguments. That is more efficient (read: faster).