使用Perl无法读取大小约为2GB的文件中的某些单词的计数

Question

我编写了一个Perl程序,它将匹配日志文件中的某些单词并将结果存储在数据库中.问题是这个程序适用于小文件,但不适用于文件大小~2GB.是否需要更改尺寸或程序？

use POSIX qw(strftime);

# load module
use DBI;

open( FILE, "/root/temp.log" ) or die "Unable to open logfile:$!\n";
$count_start   = 0;
$count_interim = 0;
$count_stop    = 0;

while (<FILE>) {
  @test = <FILE>;
  foreach $line (@test) {

    if ( $line =~ m/server start/ ) {

      #print "yes\n";
      $count_start++;
    }
    elsif ( $line =~ m/server interim-update/ ) {
      $count_stop++;
    }
    elsif ( $line =~ m/server stop/ ) {
      $count_interim++;
    }

  }
  print "$count_start\n";
  print "$count_stop\n";
  print "$count_interim\n";
  $now_string = strftime "%b %e %H:%M:%S", localtime;
  print $now_string;

  # connect
  my $dbh = DBI->connect( "DBI:Pg:dbname=postgres;host=localhost",
    "postgres", "postgres", { 'RaiseError' => 1 } );

  # execute INSERT query
  my $rows = $dbh->do(
"insert into radcount (acc,bcc,dcc) Values  ('$count_start','$count_stop','$count_interim')"
  );

  print "$rows row(s) affected\n";

  # clean up
  $dbh->disconnect();

}

close(LOG);

Answer 1

这里有一些东西 - 首先我建议改为三个arg打开你的文件句柄 - 在这里推理

open( my $fileHandle, '<', '/root/temp.log' ) or die "blah" ;

其次,你将整个文件读入一个数组 - 有一个大文件,这将占用很多内存.而是逐行阅读并处理它:

while(<$fileHandle>){
    #contents of your foreach loop
}