将二进制数据有效地读入R.

Question

从一个文本文件我正在阅读二进制数据结构如下:

0101010100101010101010101010
1010101001010101010101010111
1111101010101010100101010101

该文件有800行.每一行都是相同的长(但在文件之间有所不同,因此对其进行硬编码没有意义).我希望输入存储在一个数据框中,其中每一行都是一行,每两个数字存储在不同的列中,例如:

col1 col2 col3 col4
0      1    0    1

目前我这样做

as.matrix(read.table(text=gsub("", ' ', readLines("input"))))->g

然而,这需要太长时间,因为每行大约有70,000个0/1.

有更快的方法吗？

Answer 1

你可以pipe用awk

read.table(pipe("awk '{gsub(/./,\"& \", $1);print $1}' yourfile.txt"))
#   V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 V18 V19 V20 V21
#1  0  1  0  1  0  1  0  1  0   0   1   0   1   0   1   0   1   0   1   0   1
#2  1  0  1  0  1  0  1  0  0   1   0   1   0   1   0   1   0   1   0   1   0
#3  1  1  1  1  1  0  1  0  1   0   1   0   1   0   1   0   1   0   0   1   0
#  V22 V23 V24 V25 V26 V27 V28
#1   0   1   0   1   0   1   0
#2   1   0   1   0   1   1   1
#3   1   0   1   0   1   0   1

要么

read.table(pipe("awk '{gsub(\"\",\" \", $1);print $1}' yourfile.txt"))

fread 也可以结合使用 awk

library(data.table)
fread("awk '{gsub(/./,\"&,\", $1);print $1}' yourfile.txt")

使用与OP的数据集类似的数据集,

library(stringi)
write.table(stri_rand_strings(800,70000, '[0-1]'), file='binary1.txt',
         row.names=FALSE, quote=FALSE, col.names=FALSE)

system.time(fread("awk '{gsub(/./,\"&,\", $1);print $1}' binary1.txt"))
#  user  system elapsed 
#16.444   0.108  16.542 

Answer 2

我建议read_fwf从"阅读器"包中探索.你可以这样做:

library(readr)
len <- nchar(readLines("yourfile.txt", n = 1))
read_fwf("yourfile.txt", fwf_widths(rep(1, len)))

或者,您可以尝试"iotools"软件包,它可能更快:

library(iotools)
len <- nchar(readLines("yourfile.txt", n = 1))
input.file("yourfile.txt", formatter = dstrfw, 
            col_types = rep("integer", len), widths = rep(1, len))

这是一个小POC:

a <- tempfile()

writeLines("0101010100101010101010101010
1010101001010101010101010111
1111101010101010100101010101", a)

len <- nchar(readLines(a, n = 1))

library(readr)
read_fwf(a, fwf_widths(rep(1, len)))
# Source: local data frame [3 x 28]
# 
#   X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 X20 X21 X22 X23 X24 X25 X26 X27 X28
# 1  0  1  0  1  0  1  0  1  0   0   1   0   1   0   1   0   1   0   1   0   1   0   1   0   1   0   1   0
# 2  1  0  1  0  1  0  1  0  0   1   0   1   0   1   0   1   0   1   0   1   0   1   0   1   0   1   1   1
# 3  1  1  1  1  1  0  1  0  1   0   1   0   1   0   1   0   1   0   0   1   0   1   0   1   0   1   0   1

您的数据的维度似乎确实read_fwf令人窒息.我做了一个小测试来比较"iotools"方法和awk+ fread.

以下是示例数据:

## Creates a file named "somefile.txt"
set.seed(1)
A <- replicate(10, sample(0:1, 70000, TRUE), FALSE)
A <- sapply(A, paste, collapse = "")
writeLines(rep(A, 800/length(A)), "somefile.txt")

这是功能和结果.我已经编写了这些函数,您应该可以在实际数据上尝试它们,看看哪种方法最适合您.

显然,readr在这个阶段看起来似乎没有了:-)

Freadr <- function(infile = "somefile.txt") {
  len <- nchar(readLines(infile, n = 1))
  read_fwf(infile, fwf_widths(rep(1, len)))
}
system.time(temp1 <- Freadr())
# |===============================================================| 100%   53 MB
#    user  system elapsed 
# 466.740   0.384 466.506 

Fiotools <- function(infile = "somefile.txt") {
  len <- nchar(readLines(infile, n = 1))
  input.file(infile, formatter = dstrfw, 
             col_types = rep("integer", len), widths = rep(1, len))
}
system.time(temp2 <- Fiotools())
#    user  system elapsed 
#   7.248   0.016   7.273 

Fawk <- function(infile = "somefile.txt") {
  cmd <- sprintf("awk '{gsub(/./,\"&,\", $1);print $1}' %s", infile)
  fread(cmd)
}
system.time(temp3 <- Fawk())
#    user  system elapsed 
#  12.948   0.156  13.109 

就此而言,使用基数R也不是太糟糕:

fun4 <- function(infile = "somefile.txt") {
  do.call(rbind, lapply(strsplit(readLines(infile), "", TRUE), as.numeric))
}
system.time(fun4())
#    user  system elapsed 
#   9.056   0.260   9.304 

结果有一个matrix,所以你可能需要添加几秒钟来转换为a data.frame或者data.table如果那真的是你想要的.