Mar*_*ese 41 substring r apply lapply data.table
我使用大量固定宽度的文件(即没有分隔字符),我需要读入R.所以,通常有一个列宽度的定义来将字符串解析为变量.我可以read.fwf用来读取数据没有问题.但是,对于大文件,这可能需要很长时间.对于最近的数据集,这需要800秒来读取具有~500,000行和143个变量的数据集.
seer9 <- read.fwf("~/data/rawdata.txt",
widths = cols,
header = FALSE,
buffersize = 250000,
colClasses = "character",
stringsAsFactors = FALSE))
fread在data.tableR 中的包中解决大多数数据读取问题非常棒,除了它不解析固定宽度的文件.但是,我可以将每一行读作单个字符串(~500,000行,1列).这需要3-5秒.(我喜欢data.table.)
seer9 <- fread("~/data/rawdata.txt", colClasses = "character",
sep = "\n", header = FALSE, verbose = TRUE)
关于如何解析文本文件,有很多关于SO的好帖子.见JHoward的建议在这里,创建起始和终止列的矩阵,并substr分析数据.请参阅此处使用的GSee建议strsplit.我无法弄清楚如何使用这些数据.(另外,迈克尔史密斯对data.table邮件列表提出了一些建议,这些建议sed超出了我的实施能力.)现在,使用fread和substr()我可以在大约25-30秒内完成整个事情.请注意,在结束时强制转换为data.table需要一段时间(5秒？).
end_col <- cumsum(cols)
start_col <- end_col - cols + 1
start_end <- cbind(start_col, end_col) # matrix of start and end positions
text <- lapply(seer9, function(x) {
apply(start_end, 1, function(y) substr(x, y[1], y[2]))
})
dt <- data.table(text$V1)
setnames(dt, old = 1:ncol(dt), new = seervars)
我想知道的是这是否可以进一步改善？我知道我不是唯一一个必须读取固定宽度文件的人,所以如果能够加快速度,那么加载更大的文件(数百万行)就更容易了.我尝试使用parallelwith mclapply而data.table不是lapply,但那些没有改变任何东西.(可能由于我在R中缺乏经验)我想象一个Rcpp函数可以写得非常快,但这超出了我的技能.此外,我可能没有使用lapply并适当地应用.
我的data.table实现(带magrittr链接)需要相同的时间:
text <- seer9[ , apply(start_end, 1, function(y) substr(V1, y[1], y[2]))] %>%
data.table(.)
谁能提出建议来提高速度呢？或者这是否与它一样好？
这是在R中创建类似data.table的代码(而不是链接到实际数据).它应该有331个字符和500,000行.有一些空间可以模拟数据中缺少的字段,但这不是空格分隔的数据.(我正在阅读原始的SEER数据,以防有人感兴趣.)还包括列宽(cols)和变量名(seervars),以防这对其他人有帮助.这些是SEER数据的实际列和变量定义.
seer9 <-
data.table(rep((paste0(paste0(letters, 1000:1054, " ", collapse = ""), " ")),
500000))
cols = c(8,10,1,2,1,1,1,3,4,3,2,2,4,4,1,4,1,4,1,1,1,1,3,2,2,1,2,2,13,2,4,1,1,1,1,3,3,3,2,3,3,3,3,3,3,3,2,2,2,2,1,1,1,1,1,6,6,6,2,1,1,2,1,1,1,1,1,2,2,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,7,5,4,10,3,3,2,2,2,3,1,1,1,1,2,2,1,1,2,1,9,5,5,1,1,1,2,2,1,1,1,1,1,1,1,1,2,3,3,3,3,3,3,1,4,1,4,1,1,3,3,3,3,2,2,2,2)
seervars <- c("CASENUM", "REG", "MAR_STAT", "RACE", "ORIGIN", "NHIA", "SEX", "AGE_DX", "YR_BRTH", "PLC_BRTH", "SEQ_NUM", "DATE_mo", "DATE_yr", "SITEO2V", "LATERAL", "HISTO2V", "BEHO2V", "HISTO3V", "BEHO3V", "GRADE", "DX_CONF", "REPT_SRC", "EOD10_SZ", "EOD10_EX", "EOD10_PE", "EOD10_ND", "EOD10_PN", "EOD10_NE", "EOD13", "EOD2", "EOD4", "EODCODE", "TUMOR_1V", "TUMOR_2V", "TUMOR_3V", "CS_SIZE", "CS_EXT", "CS_NODE", "CS_METS", "CS_SSF1", "CS_SSF2", "CS_SSF3", "CS_SSF4", "CS_SSF5", "CS_SSF6", "CS_SSF25", "D_AJCC_T", "D_AJCC_N", "D_AJCC_M", "D_AJCC_S", "D_SSG77", "D_SSG00", "D_AJCC_F", "D_SSG77F", "D_SSG00F", "CSV_ORG", "CSV_DER", "CSV_CUR", "SURGPRIM", "SCOPE", "SURGOTH", "SURGNODE", "RECONST", "NO_SURG", "RADIATN", "RAD_BRN", "RAD_SURG", "SS_SURG", "SRPRIM02", "SCOPE02", "SRGOTH02", "REC_NO", "O_SITAGE", "O_SEQCON", "O_SEQLAT", "O_SURCON", "O_SITTYP", "H_BENIGN", "O_RPTSRC", "O_DFSITE", "O_LEUKDX", "O_SITBEH", "O_EODDT", "O_SITEOD", "O_SITMOR", "TYPEFUP", "AGE_REC", "SITERWHO", "ICDOTO9V", "ICDOT10V", "ICCC3WHO", "ICCC3XWHO", "BEHANAL", "HISTREC", "BRAINREC", "CS0204SCHEMA", "RAC_RECA", "RAC_RECY", "NHIAREC", "HST_STGA", "AJCC_STG", "AJ_3SEER", "SSG77", "SSG2000", "NUMPRIMS", "FIRSTPRM", "STCOUNTY", "ICD_5DIG", "CODKM", "STAT_REC", "IHS", "HIST_SSG_2000", "AYA_RECODE", "LYMPHOMA_RECODE", "DTH_CLASS", "O_DTH_CLASS", "EXTEVAL", "NODEEVAL", "METSEVAL", "INTPRIM", "ERSTATUS", "PRSTATUS", "CSSCHEMA", "CS_SSF8", "CS_SSF10", "CS_SSF11", "CS_SSF13", "CS_SSF15", "CS_SSF16", "VASINV", "SRV_TIME_MON", "SRV_TIME_MON_FLAG", "SRV_TIME_MON_PA", "SRV_TIME_MON_FLAG_PA", "INSREC_PUB", "DAJCC7T", "DAJCC7N", "DAJCC7M", "DAJCC7STG", "ADJTM_6VALUE", "ADJNM_6VALUE", "ADJM_6VALUE", "ADJAJCCSTG")
更新: LaF在原始.txt文件中进行了不到7秒的完整读取.也许有更快的方式,但我怀疑任何事情都可以做得更好.惊人的包裹.
2015年7月27日更新 只是想提供一个小的更新.我使用了新的readr包,并且我能够使用readr :: read_fwf在5秒内读取整个文件.
seer9_readr <- read_fwf("path_to_data/COLRECT.TXT",
col_positions = fwf_widths(cols))
此外,更新的stringi :: stri_sub函数至少是base :: substr()的两倍.因此,在上面的代码中使用fread来读取文件(大约4秒),然后应用于解析每一行,使用stringi :: stri_sub提取143个变量大约需要8秒,而对于base :: substr则需要19.所以,fread plus stri_sub仍然只有大约12秒的运行时间.不错.
seer9 <- fread("path_to_data/COLRECT.TXT",
colClasses = "character",
sep = "\n",
header = FALSE)
text <- seer9[ , apply(start_end, 1, function(y) substr(V1, y[1], y[2]))] %>%
data.table(.)
Jan*_*aan 30
您可以使用LaF编写的包来处理大的固定宽度文件(也太大而无法放入内存).要使用它,首先需要使用打开文件laf_open_fwf.然后,您可以像生成普通数据框一样索引生成的对象,以读取所需的数据.在下面的示例中,我读取了整个文件,但您还可以读取特定的列和/或行:
library(LaF)
laf <- laf_open_fwf("foo.dat", column_widths = cols,
column_types=rep("character", length(cols)),
column_names = seervars)
seer9 <- laf[,]
使用5000行(而不是500,000行)的示例耗时28秒,使用read.fwf1.6秒LaF.
添加使用50,000行(而不是500,000行)的示例在我的机器上使用了258秒,使用了read.fwf7秒LaF.
Mic*_*ico 29
既然(在这个和另一个关于有效读取固定宽度文件的主要问题之间)有相当多的选项可供阅读这些文件,我认为一些基准测试是合适的.
我将使用以下大面积(400 MB)文件进行比较.它只是一堆随机定义的字段和宽度的随机字符:
set.seed(21394)
wwidth = 400L
rrows = 1000000
#creating the contents at random
contents =
write.table(replicate(rrows, paste0(sample(letters, wwidth, replace = TRUE),
collapse = "")), file="testfwf.txt",
quote = FALSE, row.names = FALSE, col.names = FALSE)
#defining the fields & writing a dictionary
n_fields = 40L
endpoints = unique(c(1L, sort(sample(wwidth, n_fields - 1L)), wwidth + 1L))
cols = ist(beg = endpoints[-(n_fields + 1L)],
end = endpoints[-1L] - 1L)
dict = data.frame(column = paste0("V", seq_len(length(endpoints)) - 1L)),
start = endpoints[-length(endpoints)] - 1,
length = diff(endpoints))
write.csv(dict, file = "testdic.csv", quote = FALSE, row.names = FALSE)
我将比较这两个线程之间提到的五种方法(如果作者愿意,我会添加一些其他方法):基础版本(read.fwf),管道结果in2csv为fread(@ AnandaMahto的建议),哈德利的新readr(read_fwf),使用LaF/ ffbase(@jwijffls的建议),以及问题作者(@MarkDanese)fread与stri_subfrom 结合使用的改进(简化)版本stringi.
以下是基准测试代码:
library(data.table)
library(stringi)
library(readr)
library(LaF); library(ffbase)
library(microbenchmark)
microbenchmark(times = 5L,
utils = read.fwf("testfwf.txt", diff(endpoints), header = FALSE),
in2csv =
fread(paste("in2csv -f fixed -s",
"~/Desktop/testdic.csv",
"~/Desktop/testfwf.txt")),
readr = read_fwf("testfwf.txt", fwf_widths(diff(endpoints))),
LaF = {
my.data.laf =
laf_open_fwf('testfwf.txt', column_widths=diff(endpoints),
column_types = rep("character",
length(endpoints) - 1L))
my.data = laf_to_ffdf(my.data.laf, nrows = rrows)
as.data.frame(my.data)},
fread = fread(
"testfwf.txt", header = FALSE, sep = "\n"
)[ , lapply(seq_len(length(cols$beg)),
function(ii)
stri_sub(V1, cols$beg[ii], cols$end[ii]))])
并输出:
# Unit: seconds
# expr min lq mean median uq max neval cld
# utils 423.76786 465.39212 499.00109 501.87568 543.12382 560.84598 5 c
# in2csv 67.74065 68.56549 69.60069 70.11774 70.18746 71.39210 5 a
# readr 10.57945 11.32205 15.70224 14.89057 19.54617 22.17298 5 a
# LaF 207.56267 236.39389 239.45985 237.96155 238.28316 277.09798 5 b
# fread 14.42617 15.44693 26.09877 15.76016 20.45481 64.40581 5 a
所以它似乎readr和fread+相比stri_sub具有最快的竞争力; 内置read.fwf是明显的输家.
请注意,readr这里的真正优势是您可以预先指定列类型; 与fread你之后必须输入转换.
在@ AnandaMahto的建议中,我提供了更多选项,包括一个似乎是新赢家的选项!为了节省时间,我在新的比较中排除了上面最慢的选项.这是新代码:
library(iotools)
microbenchmark(times = 5L,
readr = read_fwf("testfwf.txt", fwf_widths(diff(endpoints))),
fread = fread(
"testfwf.txt", header = FALSE, sep = "\n"
)[ , lapply(seq_len(length(cols$beg)),
function(ii)
stri_sub(V1, cols$beg[ii], cols$end[ii]))],
iotools = input.file("testfwf.txt", formatter = dstrfw,
col_types = rep("character",
length(endpoints) - 1L),
widths = diff(endpoints)),
awk = fread(paste(
"awk -v FIELDWIDTHS='",
paste(diff(endpoints), collapse = " "),
"' -v OFS=', ' '{$1=$1 \"\"; print}' < ~/Desktop/testfwf.txt",
collapse = " "), header = FALSE))
而新的输出:
# Unit: seconds
# expr min lq mean median uq max neval cld
# readr 7.892527 8.016857 10.293371 9.527409 9.807145 16.222916 5 a
# fread 9.652377 9.696135 9.796438 9.712686 9.807830 10.113160 5 a
# iotools 5.900362 7.591847 7.438049 7.799729 7.845727 8.052579 5 a
# awk 14.440489 14.457329 14.637879 14.472836 14.666587 15.152156 5 b
因此它看起来iotools非常快且非常一致.