如何在Delphi中使用大文件？

Question

当我在内存流或文件流中使用大文件时,我看到一个"内存不足"的错误如何解决这个问题？

例:

procedure button1.cl?ck(click);
var
  mem:TMemoryStream;
  str:string;
begin
  mem:=Tmemorystream.create;
  mem.loadfromfile('test.txt');----------> there test.txt size 1 gb..
  compressstream(mem);
end;

Answer 1

你的实现非常混乱.我不确切知道CompressStream的作用,但是如果你想将一个大文件作为一个流来处理,你可以通过简单地使用TFileStream来节省内存,而不是试图一次性将整个内容读入TMemoryStream.

此外,当你完成它时,你永远不会释放TMemoryStream,这意味着你将泄漏大量的内存.(除非CompressStream负责这一点,但是代码中并不清楚,以这种方式编写它真的不是一个好主意.)

Answer 2

您无法将整个文件放入32位地址空间的单个连续块中.因此内存不足错误.

以较小的部分读取文件并逐个处理.

Answer 3

回答标题中的问题,你需要逐个字节地处理文件,如果需要的话:你肯定不会一次性将文件加载到内存中!你如何做到这一点显然取决于你需要对文件做什么; 但既然我们知道你正在尝试实现一个霍夫曼编码器,我会给你一些具体的提示.

霍夫曼编码器是流编码器:字节进入并且比特输出.输入数据的每个单元都用其相应的位模式替换.编码器不需要立即查看整个文件,因为它实际上每次只能处理一个字节.

这是你如何将文件压缩文件而不将其全部加载到内存中; 当然,没有显示实际的霍夫曼编码器,因为问题是关于使用大文件,而不是关于构建实际的编码器.这段代码包括缓冲的输入和输出,并显示了如何将实际的编码器程序链接到它.

(注意,用浏览器编写的代码;如果它不编译,你应该修复它!)

type THuffmanBuffer = array[0..1023] of Byte; // Because I need to pass the array as parameter

procedure DoActualHuffmanEncoding(const EncodeByte:Byte; var BitBuffer: THuffmanBuffer; var AtBit: Integer);
begin
  // This is where the actual Huffman encoding would happen. This procedure will
  // copy the correct encoding for EncodeByte in BitBuffer starting at AtBit bit index
  // The procedure is expected to advance the AtBit counter with the number of bits
  // that were actually written (that's why AtBit is a var parameter).   
end;

procedure HuffmanEncoder(const FileNameIn, FileNameOut: string);
var InFile, OutFile: TFileStream;
    InBuffer, OutBuffer: THuffmanBuffer;
    InBytesCount: Integer;
    OutBitPos: Integer;
    i: Integer;
begin
  // First open the InFile
  InFile := TFileStream.Create(FileNameIn, fmOpenRead or fmShareDenyWrite);
  try
    // Now prepare the OutFile
    OutFile := TFileStream.Create(FileNameOut, fmCreate);
    try
      // Start the out bit counter
      OutBitPos := 0;
      // Read from the input file, one buffer at a time (for efficiency)
      InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
      while InBytesCount <> 0 do
      begin
        // Process the input buffer byte-by-byte
        for i:=0 to InBytesCount-1 do
        begin
          DoActualHuffmanEncoding(InBuffer[i], OutBuffer, OutBitPos);
          // The function writes bits to the outer buffer, not full bytes, and the
          // encoding for a rare byte might be significantly longer then 1 byte.
          // Whenever the output buffer approaches it's capacity we'll flush it
          // out to the OutFile
          if (OutBitPos > ((SizeOf(OutBuffer)-10)*8) then
          begin
            // Ok, we've got less then 10 bytes available in the OutBuffer, time to
            // flush!
            OutFile.Write(OutBuffer, OutBitPos div 8);
            // We're now possibly left with one incomplete byte in the buffer.
            // We'll copy that byte to the start of the buffer and continue.
            OutBuffer[0] := OutBuffer[OutBitPos div 8];
            OutBitPos := OutBitPos mod 8;
          end;
        end;
        // Read next chunk
        InBytesCount := InFile.Read(InBuffer, SizeOf(InBuffer));
      end;

      // Flush the remaining of the output buffer. This time we want to flush
      // the final (potentially incomplete) byte as well, because we've got no
      // more input, there'll be no more output.
      OutFile.Write(OutBuffer, (OutBitPos + 7) div 8);

    finally OutFile.Free;
    end;     
  finally InFile.Free;
  end;
end;

霍夫曼编码器不是一个难以实现的编码器,但正确和快速地执行它可能是一个挑战.我建议你从一个正确的编码器开始,一旦你有编码和解码工作找出如何做一个快速编码器.