sra*_*rao 11 merge join r duplicates data.table
我正在尝试合并(加入)多个数据表(使用来自5个csv文件的fread获得)以形成单个数据表.当我尝试合并5个数据表时出现错误,但是当我仅合并4 MWE时工作正常:
# example data
DT1 <- data.table(x = letters[1:6], y = 10:15)
DT2 <- data.table(x = letters[1:6], y = 11:16)
DT3 <- data.table(x = letters[1:6], y = 12:17)
DT4 <- data.table(x = letters[1:6], y = 13:18)
DT5 <- data.table(x = letters[1:6], y = 14:19)
# this gives an error
Reduce(function(...) merge(..., all = TRUE, by = "x"), list(DT1, DT2, DT3, DT4, DT5))
merge.data.table(...,all = TRUE,by ="x")出错:x有一些重复的列名:yx,yy请删除或重命名副本,然后重试.
# whereas this works fine
Reduce(function(...) merge(..., all = TRUE, by = "x"), list(DT1, DT2, DT3, DT4))
x y.x y.y y.x y.y
1: a 10 11 12 13
2: b 11 12 13 14
3: c 12 13 14 15
4: d 13 14 15 16
5: e 14 15 16 17
6: f 15 16 17 18
我有一个解决方法,如果我更改DT1的第二列名称:
setnames(DT1, "y", "new_y")
# this works now
Reduce(function(...) merge(..., all = TRUE, by = "x"), list(DT1, DT2, DT3, DT4, DT5))
为什么会发生这种情况,有没有办法在不更改任何列名的情况下合并任意数量的数据表和相同的列名？
如果它只是那5个数据表(x所有数据表的位置相同),您也可以使用嵌套连接:
# set the key for each datatable to 'x'
setkey(DT1,x)
setkey(DT2,x)
setkey(DT3,x)
setkey(DT4,x)
setkey(DT5,x)
# the nested join
mergedDT1 <- DT1[DT2[DT3[DT4[DT5]]]]
或者@Frank在评论中说:
DTlist <- list(DT1,DT2,DT3,DT4,DT5)
Reduce(function(X,Y) X[Y], DTlist)
这使:
x y1 y2 y3 y4 y5
1: a 10 11 12 13 14
2: b 11 12 13 14 15
3: c 12 13 14 15 16
4: d 13 14 15 16 17
5: e 14 15 16 17 18
6: f 15 16 17 18 19
这给出了与以下相同的结果:
mergedDT2 <- Reduce(function(...) merge(..., all = TRUE, by = "x"), list(DT1, DT2, DT3, DT4, DT5))
> identical(mergedDT1,mergedDT2)
[1] TRUE
当您的x列没有相同的值时,嵌套连接将不会提供所需的解决方案:
DT1[DT2[DT3[DT4[DT5[DT6]]]]]
这给了:
x y1 y2 y3 y4 y5 y6
1: b 11 12 13 14 15 15
2: c 12 13 14 15 16 16
3: d 13 14 15 16 17 17
4: e 14 15 16 17 18 18
5: f 15 16 17 18 19 19
6: g NA NA NA NA NA 20
而:
Reduce(function(...) merge(..., all = TRUE, by = "x"), list(DT1, DT2, DT3, DT4, DT5, DT6))
得到:
x y1 y2 y3 y4 y5 y6
1: a 10 11 12 13 14 NA
2: b 11 12 13 14 15 15
3: c 12 13 14 15 16 16
4: d 13 14 15 16 17 17
5: e 14 15 16 17 18 18
6: f 15 16 17 18 19 19
7: g NA NA NA NA NA 20
使用数据:
为了使代码Reduce工作,我改变了y列的名称.
DT1 <- data.table(x = letters[1:6], y1 = 10:15)
DT2 <- data.table(x = letters[1:6], y2 = 11:16)
DT3 <- data.table(x = letters[1:6], y3 = 12:17)
DT4 <- data.table(x = letters[1:6], y4 = 13:18)
DT5 <- data.table(x = letters[1:6], y5 = 14:19)
DT6 <- data.table(x = letters[2:7], y6 = 15:20, key="x")
Reduce如果要在合并期间重命名,这里有一种保持计数器的方法:
Reduce((function() {counter = 0
function(x, y) {
counter <<- counter + 1
d = merge(x, y, all = T, by = 'x')
setnames(d, c(head(names(d), -1), paste0('y.', counter)))
}})(), list(DT1, DT2, DT3, DT4, DT5))
# x y.x y.1 y.2 y.3 y.4
#1: a 10 11 12 13 14
#2: b 11 12 13 14 15
#3: c 12 13 14 15 16
#4: d 13 14 15 16 17
#5: e 14 15 16 17 18
#6: f 15 16 17 18 19
堆栈和重塑我不认为这完全映射到merge函数但是......
mycols <- "x"
DTlist <- list(DT1,DT2,DT3,DT4,DT5)
dcast(rbindlist(DTlist,idcol=TRUE), paste0(paste0(mycols,collapse="+"),"~.id"))
# x 1 2 3 4 5
# 1: a 10 11 12 13 14
# 2: b 11 12 13 14 15
# 3: c 12 13 14 15 16
# 4: d 13 14 15 16 17
# 5: e 14 15 16 17 18
# 6: f 15 16 17 18 19
我不知道这是否会扩展到拥有比y.
合并分配
DT <- Reduce(function(...) merge(..., all = TRUE, by = mycols),
lapply(DTlist,`[.noquote`,mycols))
for (k in seq_along(DTlist)){
js = setdiff( names(DTlist[[k]]), mycols )
DT[DTlist[[k]], paste0(js,".",k) := mget(paste0("i.",js)), on=mycols, by=.EACHI]
}
# x y.1 y.2 y.3 y.4 y.5
# 1: a 10 11 12 13 14
# 2: b 11 12 13 14 15
# 3: c 12 13 14 15 16
# 4: d 13 14 15 16 17
# 5: e 14 15 16 17 18
# 6: f 15 16 17 18 19
（我不确定这是否完全扩展到其他情况。很难说，因为OP的示例确实不需要 的完整功能merge。在OP的情况下，与mycols="x"和x在所有 之间都是相同的DT*，显然合并是不合适的，正如 @eddi 所提到的。不过，一般问题很有趣，所以这就是我在这里试图解决的问题。）