在R中将大文件分割成小文件

Question

我需要将一个大文件（14 GB）分成较小的文件。该文件的格式为txt，制表符为“;” 我知道它有 70 列（字符串、双精度）。我想读取 100 万个并将它们保存在不同的文件中，file1，file2 ... fileN。

在@MKR的帮助下

但过程非常慢，我尝试使用 fread，但这是不可能的。

我该如何优化这段代码？

新代码

chunkSize <- 10000
conex <- file(description = db, open = "r")
data <- read.table(conex, nrows = chunkSize, header=T, fill=TRUE, sep =";")

index <- 0
counter <- 0
total <- 0
chunkSize <- 500000 
conex <- file(description=db,open="r")   
dataChunk <- read.table(conex, nrows=chunkSize, header=T, fill=TRUE,sep=";")

repeat {
dataChunk <- read.table(conex, nrows=chunkSize, header=FALSE, fill = TRUE, sep=";", col.names=db_colnames)
total <- total + sum(dataChunk$total)
counter <- counter + nrow(dataChunk)
write.table(dataChunk, file = paste0("MY_FILE_new",index),sep=";", row.names = FALSE)

  if (nrow(dataChunk) != chunkSize){
    print('linesok')
    break}
    index <- index + 1
  print(paste('lines', index * chunkSize))
}

Answer 1

您完全走在实现解决方案的正确轨道上。

The approach should be:

1. Read 1 million lines 
2. Write to new files
3. Read next 1 million lines
4. Write to another new files

让我们在OP尝试的行中循环转换上述逻辑：

index <- 0
counter <- 0
total <- 0
chunks <- 500000

repeat{
  dataChunk <- read.table(con, nrows=chunks, header=FALSE, fill = TRUE,                 
                          sep=";", col.names=db_colnames)

  # do processing on dataChunk (i.e adding header, converting data type) 

  # Create a new file name and write to it. You can have your own logic for file names 
  write.table(dataChunk, file = paste0("file",index))

  #check if file end has been reached and break from repeat
  if(nrow(dataChunk) < chunks){
    break
  }

  #increment the index to read next chunk
  index = index+1

}

编辑：修改为通过data.table::fread按照OP的要求读取文件来添加另一个选项。

library(data.table)

index <- 0
counter <- 0
total <- 0
chunks <- 1000000
fileName <- "myfile"

repeat{
  # With fread file is opened in each iteration
  dataChunk <- fread(input = fileName, nrows=chunks, header=FALSE, fill = TRUE,                 
                          skip = chunks*index, sep=";", col.names=db_colnames)

  # do processing on dataChunk (i.e adding header, converting data type) 

  # Create a new file name and write to it. You can have your own logic for file names
  write.table(dataChunk, file = paste0("file",index))

  #check if file end has been reached and break from repeat
  if(nrow(dataChunk) < chunks){
    break
  }

  #increment the index to read next chunk
  index = index+1

}

注意：上面的代码只是pseudo code帮助OP的部分片段。它不会自行运行并产生结果。