我想只读取文本文件每行的第一个字符,忽略其余部分.
这是一个示例文件:
x <- c(
"Afklgjsdf;bosfu09[45y94hn9igf",
"Basfgsdbsfgn",
"Cajvw58723895yubjsdw409t809t80",
"Djakfl09w50968509",
"E3434t"
)
writeLines(x, "test.txt")
我可以通过阅读所有内容readLines并使用substring获取第一个字符来解决问题:
lines <- readLines("test.txt")
substring(lines, 1, 1)
## [1] "A" "B" "C" "D" "E"
但这似乎效率低下.有没有办法说服R只读取第一个字符,而不是丢弃它们？
我怀疑应该使用一些咒语scan,但我找不到它.替代方案可能是低级文件操作(可能有seek).
由于性能仅适用于较大的文件,因此这是一个更大的测试文件,用于进行基准测试:
set.seed(2015)
nch <- sample(1:100, 1e4, replace = TRUE)
x2 <- vapply(
nch,
function(nch)
{
paste0(
sample(letters, nch, replace = TRUE),
collapse = ""
)
},
character(1)
)
writeLines(x2, "bigtest.txt")
更新:您似乎无法避免扫描整个文件.最好的速度增益似乎是使用更快的替代品readLines(Richard Scriven的stringi::stri_read_lines解决方案和Josh O'Brien的data.table::fread解决方案),或将文件视为二进制(Martin Morgan的readBin解决方案).
Ben*_*ker 20
如果允许/可以访问Unix命令行工具,则可以使用
scan(pipe("cut -c 1 test.txt"), what="", quiet=TRUE)
显然不太便携但是 大概 非常快.
使用@ RichieCotton的基准测试代码和OP建议的"bigtest.txt"文件:
expr min lq mean median uq
RC readLines 14.797830 17.083849 19.261917 18.103020 20.007341
RS read.fwf 125.113935 133.259220 148.122596 138.024203 150.528754
BB scan pipe cut 6.277267 7.027964 7.686314 7.337207 8.004137
RC readChar 1163.126377 1219.982117 1324.576432 1278.417578 1368.321464
RS scan 13.927765 14.752597 16.634288 15.274470 16.992124
Jos*_*ien 13
data.table::fread() 似乎击败了目前提出的所有解决方案,并且具有在Windows和*NIX机器上运行速度相当快的优点:
library(data.table)
substring(fread("bigtest.txt", sep="\n", header=FALSE)[[1]], 1, 1)
以下是Linux机箱上的微基准测试时序(实际上是双启动笔记本电脑,启动为Ubuntu):
Unit: milliseconds
expr min lq mean median uq max neval
RC readLines 15.830318 16.617075 18.294723 17.116666 18.959381 27.54451 100
JOB fread 5.532777 6.013432 7.225067 6.292191 7.727054 12.79815 100
RS read.fwf 111.099578 113.803053 118.844635 116.501270 123.987873 141.14975 100
BB scan pipe cut 6.583634 8.290366 9.925221 10.115399 11.013237 15.63060 100
RC readChar 1347.017408 1407.878731 1453.580001 1450.693865 1491.764668 1583.92091 100
以下是同一台笔记本电脑作为Windows机器启动的时间(使用cutRtools提供的命令行工具):
Unit: milliseconds
expr min lq mean median uq max neval cld
RC readLines 26.653266 27.493167 33.13860 28.057552 33.208309 61.72567 100 b
JOB fread 4.964205 5.343063 6.71591 5.538246 6.027024 13.54647 100 a
RS read.fwf 213.951792 217.749833 229.31050 220.793649 237.400166 287.03953 100 c
BB scan pipe cut 180.963117 263.469528 278.04720 276.138088 280.227259 387.87889 100 d
RC readChar 1505.263964 1572.132785 1646.88564 1622.410703 1688.809031 2149.10773 100 e
Mar*_*gan 13
弄清楚文件大小,将其作为单个二进制blob读取,找到感兴趣的字符的偏移量(不要计算文件末尾的最后一个'\n'),并强制转换为最终形式
f0 <- function() {
sz <- file.info("bigtest.txt")$size
what <- charToRaw("\n")
x = readBin("bigtest.txt", raw(), sz)
idx = which(x == what)
rawToChar(x[c(1L, idx[-length(idx)] + 1L)], multiple=TRUE)
}
data.table解决方案(我认为目前为止最快 - 需要将第一行包含在数据中!)
library(data.table)
f1 <- function()
substring(fread("bigtest.txt", header=FALSE)[[1]], 1, 1)
而且相比之下
> identical(f0(), f1())
[1] TRUE
> library(microbenchmark)
> microbenchmark(f0(), f1())
Unit: milliseconds
expr min lq mean median uq max neval
f0() 5.144873 5.515219 5.571327 5.547899 5.623171 5.897335 100
f1() 9.153364 9.470571 9.994560 10.162012 10.350990 11.047261 100
仍然浪费,因为整个文件在被丢弃之前被读入内存.
01/04/2015编辑将更好的解决方案带到顶端.
更新2更改scan()方法以在打开的连接上运行而不是在每次迭代时打开和关闭允许逐行读取并消除循环.时机改进了很多.
## scan() on open connection
conn <- file("bigtest.txt", "rt")
substr(scan(conn, what = "", sep = "\n", quiet = TRUE), 1, 1)
close(conn)
我还发现了stringi包中的stri_read_lines()函数,它的帮助文件说它目前是实验性的,但速度非常快.
## stringi::stri_read_lines()
library(stringi)
stri_sub(stri_read_lines("bigtest.txt"), 1, 1)
以下是这两种方法的时间安排.
## timings
library(microbenchmark)
microbenchmark(
scan = {
conn <- file("bigtest.txt", "rt")
substr(scan(conn, what = "", sep = "\n", quiet = TRUE), 1, 1)
close(conn)
},
stringi = {
stri_sub(stri_read_lines("bigtest.txt"), 1, 1)
}
)
# Unit: milliseconds
# expr min lq mean median uq max neval
# scan 50.00170 50.10403 50.55055 50.18245 50.56112 54.64646 100
# stringi 13.67069 13.74270 14.20861 13.77733 13.86348 18.31421 100
原来[慢]回答:
您可以尝试read.fwf()(固定宽度文件),将宽度设置为单个1以捕获每行上的第一个字符.
read.fwf("test.txt", 1, stringsAsFactors = FALSE)[[1L]]
# [1] "A" "B" "C" "D" "E"
当然没有经过全面测试,但适用于测试文件,并且是一个很好的函数,无需读取整个文件即可获得子字符串.
更新1: read.fwf()调用scan()和read.table()内部效率不高.我们可以跳过中间人并scan()直接尝试.
lines <- count.fields("test.txt") ## length is num of lines in file
skip <- seq_along(lines) - 1 ## set up the 'skip' arg for scan()
read <- function(n) {
ch <- scan("test.txt", what = "", nlines = 1L, skip = n, quiet=TRUE)
substr(ch, 1, 1)
}
vapply(skip, read, character(1L))
# [1] "A" "B" "C" "D" "E"
version$platform
# [1] "x86_64-pc-linux-gnu"
Windows下的每个答案的基准.
library(microbenchmark)
microbenchmark(
"RC readLines" = {
lines <- readLines("test.txt")
substring(lines, 1, 1)
},
"RS read.fwf" = read.fwf("test.txt", 1, stringsAsFactors = FALSE)$V1,
"BB scan pipe cut" = scan(pipe("cut -c 1 test.txt"),what=character()),
"RC readChar" = {
con <- file("test.txt", "r")
x <- readChar(con, 1)
while(length(ch <- readChar(con, 1)) > 0)
{
if(ch == "\n")
{
x <- c(x, readChar(con, 1))
}
}
close(con)
}
)
## Unit: microseconds
## expr min lq mean median uq
## RC readLines 561.598 712.876 830.6969 753.929 884.8865
## RS read.fwf 5079.010 6429.225 6772.2883 6837.697 7153.3905
## BB scan pipe cut 308195.548 309941.510 313476.6015 310304.412 310772.0005
## RC readChar 1238.963 1549.320 1929.4165 1612.952 1740.8300
## max neval
## 2156.896 100
## 8421.090 100
## 510185.114 100
## 26437.370 100
在更大的数据集上:
## Unit: milliseconds
## expr min lq mean median uq max neval
## RC readLines 52.212563 84.496008 96.48517 103.319789 104.124623 158.086020 20
## RS read.fwf 391.371514 660.029853 703.51134 766.867222 777.795180 799.670185 20
## BB scan pipe cut 283.442150 482.062337 516.70913 562.416766 564.680194 567.089973 20
## RC readChar 2819.343753 4338.041708 4500.98579 4743.174825 4921.148501 5089.594928 20
## RS scan 2.088749 3.643816 4.16159 4.651449 4.731706 5.375819 20