Perl vs Python日志处理性能

Question

我正在开发一个基于Web的日志管理系统,它将构建在Grails框架上,我将使用Python或Perl等文本处理语言之一.我创建了Python和Perl脚本来加载日志文件并解析每一行以将它们保存到MySQL数据库(该文件包含大约40,000行,大约7MB).使用Perl花了1分2秒,使用Python花了17秒.我曾经认为Perl比Python更快,因为Perl是原始的文本处理语言(我怀疑也来自不同的博客,我正在阅读有关Perl文本处理性能的文章).我也没想到Perl和Python之间有47秒的差异.为什么Perl需要比Python更多的时间来处理我的日志文件？是因为我使用了一些错误的数据库模块或我的代码和Perl的正则表达式可以改进吗？

注意:我是一名Java和Groovy开发人员,我没有使用Perl的经验(我使用的是Strawberry Perl v5.16).我也用Java(1分5秒)和Groovy(1分7秒)进行了这个测试,但处理日志文件的时间超过1分钟,所以两种语言都已经出来了,现在我想在Perl和蟒蛇.

PERL代码

use DBI;
use DBD::mysql;
# make connection to database
$connection = DBI->connect("dbi:mysql:logs:localhost:3306","root","") || die      "Cannot connect: $DBI::errstr";

# set the value of your SQL query
$query = "insert into logs (line_number, dated, time_stamp, thread, level, logger, user, message)
        values (?, ?, ?, ?, ?, ?, ?, ?) ";

# prepare your statement for connecting to the database
$statement = $connection->prepare($query); 

$runningTime = time;

# open text file
open (LOG,'catalina2.txt') || die "Cannot read logfile!\n";;

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}  

# close the open text file
close(LOG);

# close database connection
$connection->disconnect;

$runningTime = time - $runningTime;
printf("\n\nTotal running time: %02d:%02d:%02d\n\n", int($runningTime / 3600),   int(($runningTime % 3600) / 60), int($runningTime % 60));

# exit the script
exit;

PYTHON代码

import re
import mysql.connector
import time

file = open("D:\catalina2.txt","r")
rexp = re.compile('^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$')
conn = mysql.connector.connect(user='root',host='localhost',database='logs')
cursor = conn.cursor()

tic = time.clock()

increment  = 1
for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

conn.commit()
cursor.close()
conn.close()

toc = time.clock()
print "Total time: %s" % (toc - tic)

Answer 1

这不是一个公平的比较:

你只cursor.execute用Python 调用一次:

for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

但是你$statement->execute在Perl中多次调用:

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}  

顺便说一句,对于Python版本,cursor.execute每行调用一次会很慢.您可以使用cursor.executemany以下方法加快速度:

sql = 'insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)'
args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))

cursor.executemany(sql, args)

如果日志文件中有太多行,您可能需要将其分解为块:

args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))
    if increment % 1000 == 0:
        cursor.executemany(sql, args)
        args = []
if args:
    cursor.executemany(sql, args)    

(另外,不要使用,file.readlines()因为这会创建一个列表(可能很大).file是一个迭代器,一次吐出一行,所以for text in file就足够了.)