检查和可视化大型数据框中的间隙/空白和结构

Question

我有一个大型数据框(400000 x 50),我想直观地检查结构和空白/间隙.

是否有现有的库或ggplot2函数,可以吐出这样的图片:

其中红色可能是"日期",蓝色代表"因子",绿色代表"字符",黑色代表空白/ NA.

Answer 1

您是否尝试过dfviewr的lasagnar？以下内容将再现df.in包中50行x 10列的所需图形:

library(devtools)
install_github("swihart/lasagnar")
library(lasagnar)   
dfviewr(df=df.in)
## also try:
##dfviewr(df=df.in, legend=FALSE)
##dfviewr(df=df.in, gridlines=FALSE)

所以,公平地说,dfviewr在问题发生时并不存在,但要看到导致其发展的一些想法以及如何实际可视化400,000行,请参阅最底层的for循环,并且不要太莽撞并且运行功能df2.in(400,000 x 50):

## Do not run:
## system.time(dfviewr(df=df2.in, gridlines=FALSE)) ## 10 minutes before useRaster=TRUE                                          
                                                    ##  2 minutes after

此外,tabplot:::tableplot()似乎不支持日期或字符:

library(tabplot)
tableplot(df.in)

生产:

Error in ff(initdata = initdata, length = length, levels = levels, ordered = ordered, : vmode 'character' not implemented

所以我们删除了字符列(#9):

tableplot(df.in[,c(-9)])

产生:

Error in UseMethod("as.hi") : no applicable method for 'as.hi' applied to an object of class "c('POSIXct', 'POSIXt')"

所以我们也删除了第一列(日期):

tableplot(df.in[,c(-1,-9)])

得到

对于df2.in没有日期或字符列的400,000 x 50 ,图像渲染非常快(6秒):

system.time(tableplot(df2.in[,c(-(1+seq(0,40,10)), -(9+seq(0,40,10))) ]))

感兴趣的读者......

我首先介绍50行的婴儿示例,然后是400,000行的示例.

对于它的价值,我在@cmbarbu的评论中,关于在同一个地块上直观地观察400K行的评论受到屏幕的限制,该屏幕最多具有2K像素的高度,因此在页面之间的某种分离可能有利于防止过度绘图.我通过在1000个图表/页面中制作包含400行的PDF文档来尝试将此分开.

我不知道将使用data.frame作为输入呈现请求的绘图的函数.我的方法将生成data.frame的矩阵掩码,然后lasagna()从lasagnargithub上的包中使用. lasagna()是函数的包装器,image( t(X)[, (nrow(X):1)] )其中X是矩阵.此调用重新排序行,以便它们匹配data.frame的顺序,并且包装器允许切换网格线和添加图例的功能(legend = TRUE将调用image.plot( t(X)[, (nrow(X):1)] )- 但是,在下面的示例中,我明确添加了一个图例而不是使用image.plot()).

任务的库

library(fields)
library(colorspace)  
library(lubridate)
library(devtools)
install_github("swihart/lasagnar")
library(lasagnar)   

创建一个50行的示例数据帧(在400K示例之前的宝贝示例)

df.in <- data.frame(date=seq(ymd('2012-04-07'),ymd('2013-03-22'), 
                    by = '1 week'),
           col1=rnorm(50),
           col2=rnorm(50),
           col3=rnorm(50),
           col4=rnorm(50),
           col5=as.factor(c("A","B")),
           col6=as.factor(c("MS","PHD")),
           col7=rnorm(50),
           col8=(c("cherlene","randy")),
           col9=rnorm(50),
           stringsAsFactors=FALSE)

诱发失踪

df.in[19:23  , 2:4  ] <- NA
df.in[c(7, 9),      ] <- NA
df.in[2:30   , 4    ] <- NA
df.in[10     , 7    ] <- NA
df.in[14     , 6:10 ] <- NA

检查结构

str(df.in)

准备掩模矩阵

mat.out <- matrix(NA, nrow=nrow(df.in), ncol=ncol(df.in))

然后循环遍历列的类型; 最后申请is.na()

## red for dates
mat.out[,sapply(df.in,is.POSIXct)] <- 1
## blue for factors
mat.out[,sapply(df.in,is.factor)] <- 2
## green for characters
mat.out[,sapply(df.in,is.character)] <- 3
## white for numeric
mat.out[,sapply(df.in,is.numeric)] <- 4
## black for NA
mat.out[is.na(df.in)] <- 5

行名称可能很适合追溯到原始数据

row.names(mat.out) <- 1:nrow(df.in)

render {lasagna(X)是图像的包装器(t(X)[,(nrow(X):1)])}

lasagna(mat.out, col=c("red","blue","green","white","black"), 
        cex=0.67, main="")

传说是可能的:

lasagna(mat.out, col=c("red","blue","green","white","black"), 
        cex=.67, main="")
legend("bottom", fill=c("red","blue","green","white","black"),
        legend=c("dates", "factors", "characters", "numeric", "NA"), 
        horiz=T, xpd=NA, inset=c(-.15), border="black")

用网格线关闭网格线= FALSE

lasagna(mat.out, col=c("red","blue","green","white","black"), 
        cex=.67, main="", gridlines=FALSE)
legend("bottom", fill=c("red","blue","green","white","black"),
        legend=c("dates", "factors", "characters", "numeric", "NA"), 
        horiz=T, xpd=NA, inset=c(-.15), border="black")

让我们举一个OP数据大小的例子:400,000行x 50 cols

创建一个示例数据帧

df2.10 <- data.frame(date=seq(ymd('2012-04-07'),ymd('2013-03-22'), 
                    by = '1 week'),
           col1=rnorm(400000),
           col2=rnorm(400000),
           col3=rnorm(400000),
           col4=rnorm(400000),
           col5=as.factor(c("A","B")),
           col6=as.factor(c("MS","PHD")),
           col7=rnorm(400000),
           col8=(c("cherlene","randy")),
           col9=rnorm(400000),
           stringsAsFactors=FALSE)

诱发失踪

df2.10[c(19:23), c(2:4)  ] <- NA
df2.10[c(7,  9),         ] <- NA
df2.10[c(2:30), 4        ] <- NA
df2.10[10     , 7        ] <- NA
df2.10[14     , c(6:10)  ] <- NA    
df2.10[c(450:750), ] <- NA
df2.10[c(399990:399999), ] <- NA

cbind成50列宽df; 检查结构

df2.in <- cbind(df2.10, df2.10, df2.10, df2.10, df2.10)
str(df2.in)

准备掩模矩阵

mat.out <- matrix(NA, nrow=nrow(df2.in), ncol=ncol(df2.in))

然后循环遍历列的类型; 最后申请is.na()

## red for dates
mat.out[,sapply(df2.in,is.POSIXct)] <- 1
## blue for factors
mat.out[,sapply(df2.in,is.factor)] <- 2
## green for characters
mat.out[,sapply(df2.in,is.character)] <- 3
## white for numeric
mat.out[,sapply(df2.in,is.numeric)] <- 4
## black for NA
mat.out[is.na(df2.in)] <- 5

行名称可能很适合追溯到原始数据

row.names(mat.out) <- 1:nrow(df2.in)

渲染{lasagna_plain(X)没有gridelines或rownames}

pdf("pages1000.pdf")
  system.time(
    for(i in 1:1000){
        lasagna_plain(mat.out[((i-1)*400+1):(400*i),],
                      col=c("red","blue","green","white","black"), cex=1, 
                      main=paste0("rows: ", (i-1)*400+1,  " - ",  (400*i)))
    }
  )
dev.off()

for循环在我的机器上完成了40秒,之后很快就完成了PDF.现在只需在PDF查看器中标准化页面大小后向下翻页,查看以下页面/图: