以合理,安全和有效的方式复制文件

Question

我搜索一个复制文件(二进制或文本)的好方法.我写过几个样本,每个人都在工作.但我想听听经验丰富的程序员的意见.

我错过了很好的例子并搜索了一种适用于C++的方法.

ANSI-C-WAY

#include <iostream>
#include <cstdio>    // fopen, fclose, fread, fwrite, BUFSIZ
#include <ctime>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    // BUFSIZE default is 8192 bytes
    // BUFSIZE of 1 means one chareter at time
    // good values should fit to blocksize, like 1024 or 4096
    // higher values reduce number of system calls
    // size_t BUFFER_SIZE = 4096;

    char buf[BUFSIZ];
    size_t size;

    FILE* source = fopen("from.ogv", "rb");
    FILE* dest = fopen("to.ogv", "wb");

    // clean and more secure
    // feof(FILE* stream) returns non-zero if the end of file indicator for stream is set

    while (size = fread(buf, 1, BUFSIZ, source)) {
        fwrite(buf, 1, size, dest);
    }

    fclose(source);
    fclose(dest);

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " << end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

POSIX-WAY(K&R在"C编程语言"中使用它,更低级别)

#include <iostream>
#include <fcntl.h>   // open
#include <unistd.h>  // read, write, close
#include <cstdio>    // BUFSIZ
#include <ctime>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    // BUFSIZE defaults to 8192
    // BUFSIZE of 1 means one chareter at time
    // good values should fit to blocksize, like 1024 or 4096
    // higher values reduce number of system calls
    // size_t BUFFER_SIZE = 4096;

    char buf[BUFSIZ];
    size_t size;

    int source = open("from.ogv", O_RDONLY, 0);
    int dest = open("to.ogv", O_WRONLY | O_CREAT /*| O_TRUNC/**/, 0644);

    while ((size = read(source, buf, BUFSIZ)) > 0) {
        write(dest, buf, size);
    }

    close(source);
    close(dest);

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " << end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

KISS-C++ - Streambuffer-WAY

#include <iostream>
#include <fstream>
#include <ctime>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    ifstream source("from.ogv", ios::binary);
    ofstream dest("to.ogv", ios::binary);

    dest << source.rdbuf();

    source.close();
    dest.close();

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " <<  end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

COPY-算法-C++ - WAY

#include <iostream>
#include <fstream>
#include <ctime>
#include <algorithm>
#include <iterator>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    ifstream source("from.ogv", ios::binary);
    ofstream dest("to.ogv", ios::binary);

    istreambuf_iterator<char> begin_source(source);
    istreambuf_iterator<char> end_source;
    ostreambuf_iterator<char> begin_dest(dest); 
    copy(begin_source, end_source, begin_dest);

    source.close();
    dest.close();

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " <<  end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

OWN-BUFFER-C++ - WAY

#include <iostream>
#include <fstream>
#include <ctime>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    ifstream source("from.ogv", ios::binary);
    ofstream dest("to.ogv", ios::binary);

    // file size
    source.seekg(0, ios::end);
    ifstream::pos_type size = source.tellg();
    source.seekg(0);
    // allocate memory for buffer
    char* buffer = new char[size];

    // copy file    
    source.read(buffer, size);
    dest.write(buffer, size);

    // clean up
    delete[] buffer;
    source.close();
    dest.close();

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " <<  end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

LINUX-WAY //需要内核> = 2.6.33

#include <iostream>
#include <sys/sendfile.h>  // sendfile
#include <fcntl.h>         // open
#include <unistd.h>        // close
#include <sys/stat.h>      // fstat
#include <sys/types.h>     // fstat
#include <ctime>
using namespace std;

int main() {
    clock_t start, end;
    start = clock();

    int source = open("from.ogv", O_RDONLY, 0);
    int dest = open("to.ogv", O_WRONLY | O_CREAT /*| O_TRUNC/**/, 0644);

    // struct required, rationale: function stat() exists also
    struct stat stat_source;
    fstat(source, &stat_source);

    sendfile(dest, source, 0, stat_source.st_size);

    close(source);
    close(dest);

    end = clock();

    cout << "CLOCKS_PER_SEC " << CLOCKS_PER_SEC << "\n";
    cout << "CPU-TIME START " << start << "\n";
    cout << "CPU-TIME END " << end << "\n";
    cout << "CPU-TIME END - START " <<  end - start << "\n";
    cout << "TIME(SEC) " << static_cast<double>(end - start) / CLOCKS_PER_SEC << "\n";

    return 0;
}

环境

• GNU/LINUX(Archlinux)
• 内核3.3
• GLIBC-2.15,LIBSTDC++ 4.7(GCC-LIBS),GCC 4.7,Coreutils 8.16
• 使用RUNLEVEL 3(多用户,网络,终端,无GUI)
• INTEL SSD-Postville 80 GB,最高可达50%
• 复制270 MB OGG-VIDEO-FILE

重现步骤

 1. $ rm from.ogg
 2. $ reboot                           # kernel and filesystem buffers are in regular
 3. $ (time ./program) &>> report.txt  # executes program, redirects output of program and append to file
 4. $ sha256sum *.ogv                  # checksum
 5. $ rm to.ogg                        # remove copy, but no sync, kernel and fileystem buffers are used
 6. $ (time ./program) &>> report.txt  # executes program, redirects output of program and append to file

结果(使用的CPU时间)

Program  Description                 UNBUFFERED|BUFFERED
ANSI C   (fread/frwite)                 490,000|260,000  
POSIX    (K&R, read/write)              450,000|230,000  
FSTREAM  (KISS, Streambuffer)           500,000|270,000 
FSTREAM  (Algorithm, copy)              500,000|270,000
FSTREAM  (OWN-BUFFER)                   500,000|340,000  
SENDFILE (native LINUX, sendfile)       410,000|200,000  

Filesize不会改变.
sha256sum打印相同的结果.
视频文件仍然可以播放.

问题

• 你更喜欢哪种方法？
• 你知道更好的解决方案吗？
• 你看到我的代码中有任何错误吗？

• 你知道避免解决方案的理由吗？

• FSTREAM(KISS,Streambuffer)
  我非常喜欢这个,因为它非常简短.到目前为止我知道运算符<<为rdbuf()重载并且没有转换任何东西.正确？

谢谢

更新1
我以这种方式更改了所有样本中的源,文件描述符的打开和关闭包含在clock()的测量中.它们在源代码中没有其他重大变化.结果没有改变!我也花时间仔细检查我的结果.

更新2
ANSI C样本已更改:while循环的条件不再调用feof()而是将fread()移动到条件中.看起来,代码现在运行速度提高了10,000个时钟.

测量改变了:前面的结果总是被缓冲,因为我重复了几次每个程序的旧命令行rm to.ogv && sync && time ./program.现在我为每个程序重启系统.无缓冲的结果是新的,毫不奇怪.无缓冲的结果并没有真正改变.

如果我不删除旧副本,程序会有不同的反应.使用POSIX和SENDFILE 覆盖现有缓冲的文件更快,所有其他程序都更慢.也许截断或创建选项会对此行为产生影响.但是用相同的副本覆盖现有文件并不是真实的用例.

使用cp执行复制需要0.44秒无缓冲和0.30秒缓冲.所以cp比POSIX样本慢一点.对我来说很好看.

也许我还添加了mmap()和copy_file()boost :: filesystem的示例和结果.

更新3
我也把它放在博客页面上并稍微扩展了一下.包括splice(),它是Linux内核的低级函数.也许会有更多带有Java的样本. http://www.ttyhoney.com/blog/?page_id=69

Answer 1

以理智的方式复制文件:

#include <fstream>

int main()
{
    std::ifstream  src("from.ogv", std::ios::binary);
    std::ofstream  dst("to.ogv",   std::ios::binary);

    dst << src.rdbuf();
}

这是如此简单和直观,阅读它是值得的额外费用.如果我们做了很多,最好还是回到对文件系统的OS调用.我确信boost它的文件系统类中有一个复制文件方法.

有一种与文件系统交互的C方法:

#include <copyfile.h>

int
copyfile(const char *from, const char *to, copyfile_state_t state, copyfile_flags_t flags);

Answer 2

使用C++ 17,复制文件的标准方法是包括<filesystem>标题并使用:

bool copy_file( const std::filesystem::path& from,
                const std::filesystem::path& to);

bool copy_file( const std::filesystem::path& from,
                const std::filesystem::path& to,
                std::filesystem::copy_options options);

第一种形式相当于第二种形式,copy_options::none用作选项(另见copy_file).

该filesystem库最初是作为boost.filesystemC++ 17 开发并最终合并到ISO C++中的.

Answer 3

太多!

"ANSI C"方式缓冲区是冗余的,因为a FILE已经被缓冲.(此内部缓冲区的大小是BUFSIZ实际定义的大小.)

"OWN-BUFFER-C++-WAY"将会很慢,它会进行fstream大量的虚拟调度,并再次维护内部缓冲区或每个流对象.("COPY-ALGORITHM-C++ - WAY"不会受此影响,因为streambuf_iterator该类会绕过流层.)

我更喜欢"COPY-ALGORITHM-C++ - WAY",但是如果没有构建fstream,只std::filebuf需要在不需要实际格式化的情况下创建裸实例.

对于原始性能,您无法击败POSIX文件描述符.它在任何平台上都很丑陋但便携且快速.

Linux的方式似乎非常快 - 也许操作系统在I/O完成之前让函数返回？在任何情况下,这对于许多应用来说都不够便携.

编辑:啊,"原生Linux"可能通过使用异步I/O交错读写来提高性能.让命令堆积可以帮助磁盘驱动器决定何时最好寻找.您可以尝试使用Boost Asio或pthreads进行比较.至于"无法击败POSIX文件描述符"......如果您对数据做任何事情,那就是真的,而不仅仅是盲目复制.

Answer 4

我想提出一个非常重要的注意事项,即使用sendfile()的LINUX方法存在一个主要问题,即它无法复制大小超过2GB的文件!我已经按照这个问题实现了它并且遇到了问题,因为我正在使用它来复制大小为GB的HDF5文件.

http://man7.org/linux/man-pages/man2/sendfile.2.html

sendfile()最多传输0x7ffff000(2,147,479,552)个字节,返回实际传输的字节数.(在32位和64位系统上都是如此.)