快速将矢量写入文件r

Question

将向量写入文件的最快方法是什么？我有一个大约200万行的字符向量,它有相当大的值(200个字符).我现在正在做

write(myVector, "myFile.txt")

但这非常缓慢.我一直在寻找解决方案,但快速写入功能(例如fwrite)仅将数据帧/矩阵作为输入.谢谢!

Answer 1

在尝试了几个选项之后,我找到了最快的选择data.table::fwrite.就像@Gregor在他的第一个评论中所说的那样,它的速度提高了一个数量级,值得加载额外的包.它也是产生更大文件的产品之一.(另一个是readr::write_lines.感谢Calum You的评论,我忘记了这个.)

library(data.table)
library(readr)

set.seed(1)    # make the results reproducible
n <- 1e6
x <- rnorm(n)

t1 <- system.time({
    sink(file = "test_sink.txt")
    cat(x, "\n")
    sink()
})
t2 <- system.time({
    cat(x, "\n", file = "test_cat.txt")
})
t3 <- system.time({
    write(x, file = "test_write.txt")
})
t4 <- system.time({
    fwrite(list(x), file = "test_fwrite.txt")
})
t5 <- system.time({
    write_lines(x, "test_write_lines.txt")
})

rbind(sink = t1[1:3], cat = t2[1:3], 
      write = t3[1:3], fwrite = t4[1:3],
      readr = t5[1:3])
#       user.self sys.self elapsed
#sink        4.18    11.64   15.96
#cat         3.70     4.80    8.57
#write       3.71     4.87    8.64
#fwrite      0.42     0.02    0.51
#readr       2.37     0.03    6.66

在他的第二个评论中,格雷戈尔注意到as.list并且list行为不同.差异很重要.前者将向量写为一行和多列,后者写入一列和多行.

速度差异也很明显:

fw1 <- system.time({
    fwrite(as.list(x), file = "test_fwrite.txt")
})
fw2 <- system.time({
    fwrite(list(x), file = "test_fwrite2.txt")
})

rbind(as.list = fw1[1:3], list = fw2[1:3])
#        user.self sys.self elapsed
#as.list      0.67     0.00    0.75
#list         0.19     0.03    0.11

最后清理.

unlink(c("test_sink.txt", "test_cat.txt", "test_write.txt",
         "test_fwrite.txt", "test_fwrite2.txt", "test_write_lines.txt"))

Answer 2

我发现writeBin速度是原来的两倍fwrite.试试这个:

 zz <- file("myFile.txt", "wb")
 writeBin( paste(myVector, collapse="\n"), zz ) 
  close(zz)

使用Rui提供的相同计时方法(旧盒子):

            user.self sys.self elapsed
sink            9.650    7.900  17.418
cat             6.507    7.870  14.254
write           6.436    7.849  14.171
fwrite          0.500    0.051   0.593
write_lines     4.337    0.150   4.451
writeBin        0.238    0.006   0.242