Python cStringIO在写入时比StringIO花费更多时间(字符串方法的性能)

Question

在我的方式配置python中的字符串方法,以便我可以使用最快的方法.我有这个代码来测试文件中的字符串连接,StringIO,StringIO和普通字符串.

#!/usr/bin/env python
#title           : pythonTiming.py
#description     : Will be used to test timing function in python
#author          : myusuf
#date            : 19-11-2014
#version         : 0
#usage           :python pythonTiming.py
#notes           :
#python_version  :2.6.6  
#==============================================================================

import time
import cStringIO
import StringIO

class Timer(object):

    def __enter__(self):
        self.start = time.time()
        return self

    def __exit__(self, *args):
        self.end = time.time()
        self.interval = self.end - self.start

testbuf = """ Hello This is a General String that will be repreated
This string will be written to a file , StringIO and a sregualr strin then see the best to handle string according to time 

""" * 1000

MyFile = open("./testfile.txt" ,"wb+")
MyStr  = ''
MyStrIo = StringIO.StringIO()
MycStrIo = cStringIO.StringIO()

def strWithFiles():
    global MyFile
    print "writing string to file "
    for index in range(1000):
        MyFile.write(testbuf) 
    pass

def strWithStringIO():
    global MyStrIo
    print "writing string to StrinIO "
    for index in range(1000):
        MyStrIo.write(testbuf)

def strWithStr():
    global MyStr
    print "Writing String to STR "
    for index in range(500):
        MyStr =  MyStr +  testbuf

def strWithCstr():
    global MycStrIo
    print "writing String to Cstring"
    for index in range(1000):
        MycStrIo.write(testbuf)

with Timer() as t:
    strWithFiles()
print('##Request took %.03f sec.' % t.interval)

with Timer() as t:                                                                                
    strWithStringIO()
print('###Request took %.03f sec.' % t.interval)  

with Timer() as t:                                                                                
    strWithCstr()
print('####Request took %.03f sec.' % t.interval)  

with Timer() as t:
    read1 = 'x' + MyFile.read(-1)
print('file read ##Request took %.03f sec.' % t.interval)

with Timer() as t:
    read2 = 'x' + MyStrIo.read(-1)
print('stringIo read ###Request took %.03f sec.' % t.interval)

with Timer() as t:
    read3 = 'x' + MycStrIo.read(-1)
print('CString read ####Request took %.03f sec.' % t.interval)




MyFile.close()

1. 虽然Python文档站点说这cStringIO比速度快StringIO但结果表明StringIO串联性能更好,为什么呢？

2. 另一方面,读取cStringIO速度比StringIO(它的行为类似于文件)更快,因为我读取文件的实现并且cStringIO在C中,所以为什么字符串连接很慢？

3. 有没有其他方法比这些方法更快地处理字符串？

Answer 1

StringIO表现更好的原因是幕后它只保留已写入其中的所有字符串的列表,并且只在必要时才组合它们.因此,写操作就像将对象附加到列表一样简单.但是,该cStringIO模块没有这种奢侈品,必须将每个字符串的数据复制到其缓冲区中,并在必要时调整其缓冲区大小(这会在写入大量数据时产生大量冗余的数据复制).

由于您要编写大量较大的字符串,这意味着与之相比,StringIO可以做的工作较少cStringIO.从StringIO您写入的对象读取时,它可以通过计算写入其中的字符串长度总和来预先选择该大小的缓冲区,从而优化所需的复制量.

但是,StringIO这不是加入一系列字符串的最快方法.这是因为它提供了额外的功能(寻找缓冲区的不同部分并在那里写入数据).如果不需要此功能,您只需将列表字符串连接在一起,那么这str.join是执行此操作的最快方法.

joined_string = "".join(testbuf for index in range(1000))
# or building the list of strings to join separately
strings = []
for i in range(1000):
    strings.append(testbuf)
joined_string = "".join(strings)