Python:如何使用生成器来避免sql内存问题

Question

我有以下访问mysql数据库的方法,并且在我无权访问的服务器中执行查询以改变有关增加内存的任何内容.我是发电机的新手,并开始阅读更多关于它的信息,并认为我可以将其转换为使用发电机.

def getUNames(self):
    globalUserQuery = ur'''SELECT gu_name FROM globaluser WHERE gu_locked = 0'''
    global_user_list = []
    try:
        self.gdbCursor.execute(globalUserQuery)
        rows = self.gdbCursor.fetchall()
        for row in rows:
            uName = unicode(row['gu_name'], 'utf-8')
            global_user_list.append(uName)
        return global_user_list
    except Exception, e:
        traceback.print_exc()

我使用此代码如下:

for user_name in getUNames():
...

这是我从服务器端获得的错误:

^GOut of memory (Needed 725528 bytes)
Traceback (most recent call last):
...
packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (2008, 'MySQL client ran out of memory')

我应该如何使用生成器来避免这种情况:

while true:
   self.gdbCursor.execute(globalUserQuery)
   row = self.gdbCursor.fetchone()
   if row is None: break
   yield row

不确定上述是否是正确的方法,因为我期待一个列表作为我的数据库方法的结果.我认为最好的是从查询获取块并返回一个列表,一旦完成,生成器只要查询返回结果就会给出下一个集合.

Answer 1

使用MySQLdb,默认游标在调用时将整个结果集加载到Python列表中cursor.execute(..).对于可能导致MemoryError的大型查询,无论您是否使用生成器.

相反,使用SSCursor或SSDictCursor.这些将结果集保留在服务器端,并允许您通过客户端结果集中的项目进行交互:

import MySQLdb  
import MySQLdb.cursors as cursors
import traceback

def getUNames(self):
    # You may of course want to define `self.gdbCursor` somewhere else...
    conn = MySQLdb.connect(..., cursorclass=cursors.SSCursor)
    #                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    #                       Set the cursor class to SSCursor here
    self.gdbCursor = conn.cursor()

    globalUserQuery = ur'''SELECT gu_name FROM globaluser WHERE gu_locked = 0'''
    try:
        self.gdbCursor.execute(globalUserQuery)
        for row in self.gdbCursor:
            uName = unicode(row['gu_name'], 'utf-8')
            yield uName
    except Exception as e:
        traceback.print_exc()

没有太多关于默认Cursor和默认值之间差异的文档SSCursor.我知道的最好的来源是Cursor Mixin类本身的文档字符串:

默认光标使用CursorStoreResultMixIn:

In [2]: import MySQLdb.cursors as cursors
In [8]: print(cursors.CursorStoreResultMixIn.__doc__)
This is a MixIn class which causes the entire result set to be
    stored on the client side, i.e. it uses mysql_store_result(). If the
    result set can be very large, consider adding a LIMIT clause to your
    query, or using CursorUseResultMixIn instead.

而SSCursor使用CursorUseResultMixIn:

In [9]: print(cursors.CursorUseResultMixIn.__doc__)
This is a MixIn class which causes the result set to be stored
    in the server and sent row-by-row to client side, i.e. it uses
    mysql_use_result(). You MUST retrieve the entire result set and
    close() the cursor before additional queries can be peformed on
    the connection.

因为我改成getUNames了一个生成器,它会像这样使用:

for row in self.getUnames():
    ...