用"超大"矩阵绘制热图

Question

我想画一张热图.
我有100k*100k方阵(50Gb(csv),右上方有数字,0上有0).

我想问一下"如何用R绘制热图？" 这个巨大的数据集.
我正在尝试在大型RAM机器上使用此代码.

d = read.table("data.csv", sep=",")
d = as.matrix(d + t(d))
heatmap(d)

我尝试了一些像heatmap.2(在gplots中)或其他类似的库.
但他们需要花费很多时间和回忆.

Answer 1

我建议你在绘制热图之前对矩阵进行大量下采样,例如,对每个子矩阵进行均值处理(如@IaroslavDomin所示):

# example of big mx 10k x 10 k
bigMx <- matrix(rnorm(10000*10000,mean=0,sd=100),10000,10000)

# here we downsample the big matrix 10k x 10k to 100x100
# by averaging each submatrix
downSampledMx <- matrix(NA,100,100)
subMxSide <- nrow(bigMx)/nrow(downSampledMx)
for(i in 1:nrow(downSampledMx)){
  rowIdxs <- ((subMxSide*(i-1)):(subMxSide*i-1))+1
  for(j in 1:ncol(downSampledMx)){
    colIdxs <- ((subMxSide*(j-1)):(subMxSide*j-1))+1
    downSampledMx[i,j] <- mean(bigMx[rowIdxs,colIdxs])
  }
}

# NA to disable the dendrograms
heatmap(downSampledMx,Rowv=NA,Colv=NA) 

对于你的庞大矩阵肯定需要一段时间来计算downSampledMx,但这应该是可行的.

编辑:

我认为下采样应保留可识别的"宏模式",例如,请参阅以下示例:

# create a matrix with some recognizable pattern
set.seed(123)
bigMx <- matrix(rnorm(50*50,mean=0,sd=100),50,50)
diag(bigMx) <- max(bigMx) # set maximum value on the diagonal
# set maximum value on a circle centered on the middle
for(i in 1:nrow(bigMx)){
  for(j in 1:ncol(bigMx)){
    if(abs((i - 25)^2 + (j - 25)^2 - 10^2) <= 16)
      bigMx[i,j] <- max(bigMx)
  }
}

# plot the original heatmap
heatmap(bigMx,Rowv=NA,Colv=NA, main="original")


# function used to down sample
downSample <- function(m,newSize){
  downSampledMx <- matrix(NA,newSize,newSize)
  subMxSide <- nrow(m)/nrow(downSampledMx)
  for(i in 1:nrow(downSampledMx)){
    rowIdxs <- ((subMxSide*(i-1)):(subMxSide*i-1))+1
    for(j in 1:ncol(downSampledMx)){
      colIdxs <- ((subMxSide*(j-1)):(subMxSide*j-1))+1
      downSampledMx[i,j] <- mean(m[rowIdxs,colIdxs])
    }
  }
  return(downSampledMx)
}

# downsample x 2 and plot heatmap
downSampledMx <- downSample(bigMx,25)
heatmap(downSampledMx,Rowv=NA,Colv=NA, main="downsample x 2") 

# downsample x 5 and plot heatmap
downSampledMx <- downSample(bigMx,10)
heatmap(downSampledMx,Rowv=NA,Colv=NA, main="downsample x 5") 

这是3个热图: