共享:内存:使用sqlite3包在python中的不同线程之间的数据库

Question

我想在python中创建一个:memory:database,并从不同的线程访问它.基本上是这样的:

class T(threading.Thread):
    def run(self):
        self.conn = sqlite3.connect(':memory:')
        # do stuff with the database

for i in xrange(N):
    T().start()

并让所有连接引用同一个数据库.

我知道传递check_same_thread=True给connect函数并在线程之间共享连接,但是如果可能的话,我希望避免这样做.谢谢你的帮助.

编辑:纠正错字.我最初说"将所有连接引用到同一个线程"将线程替换为数据库.

Answer 1

SQLite在过去4年中有所改进,因此现在可以共享内存数据库.检查以下代码:

import sqlite3

foobar_uri = 'file:foobar_database?mode=memory&cache=shared'
not_really_foobar_uri = 'file:not_really_foobar?mode=memory&cache=shared'

# connect to databases in no particular order
db2 = sqlite3.connect(foobar_uri, uri=True)
db_lol = sqlite3.connect(not_really_foobar_uri, uri=True)
db1 = sqlite3.connect(foobar_uri, uri=True)

# create cursor as db2
cur2 = db2.cursor()

# create table as db2
db2.execute('CREATE TABLE foo (NUMBER bar)')

# insert values as db1
db1.execute('INSERT INTO foo VALUES (42)')
db1.commit()

# and fetch them from db2 through cur2
cur2.execute('SELECT * FROM foo')
print(cur2.fetchone()[0])  # 42

# test that db_lol is not shared with db1 and db2
try:
    db_lol.cursor().execute('SELECT * FROM foo')
except sqlite3.OperationalError as exc:
    print(exc)  # just as expected

故意纠缠数据库访问,以显示从SQLite的角度来看,具有相同名称的内存数据库的两个连接是相同的.

参考文献:

1. SQLite URI
2. SQLite共享缓存

不幸的是,URI的连接仅在Python 3.4之后可用.但是,如果你有Python 2.6或更高版本(但不是Python 3),内置sqlite3模块仍然能够导入APSW连接,这可以用来实现相同的效果.这里是插入式sqlite3模块更换:

from sqlite3 import *
from sqlite3 import connect as _connect
from apsw import Connection as _ApswConnection
from apsw import SQLITE_OPEN_READWRITE as _SQLITE_OPEN_READWRITE
from apsw import SQLITE_OPEN_CREATE as _SQLITE_OPEN_CREATE
from apsw import SQLITE_OPEN_URI as _SQLITE_OPEN_URI

# APSW and pysqlite use different instances of sqlite3 library, so initializing
# APSW won't help pysqlite. Because pysqlite does not expose any way to
# explicitly call sqlite3_initialize(), here goes an ugly hack. This only has
# to be done once per process.
_connect(':memory:').close()

def connect(database, timeout=5.0, detect_types=0, isolation_level=None,
            check_same_thread=True, factory=Connection, cached_statements=100,
            uri=False):
    flags = _SQLITE_OPEN_READWRITE | _SQLITE_OPEN_CREATE

    if uri:
        flags |= _SQLITE_OPEN_URI

    db = _ApswConnection(database, flags, None, cached_statements)
    conn = _connect(db, timeout, detect_types, isolation_level, 
                    check_same_thread, factory, cached_statements)

    return conn

Answer 2

没有黑客攻击sqlite3库本身就无法重用:memory:数据库,因为它保证对每个连接都是独占的和私有的.要破解对它的访问,请仔细查看src/pager.csqlite3发行版(而不是Python模块发行版).也许,最便捷的方式来实现,这将是令:memory:00,:memory:something,:memory:okay_hai等别名,以满足不同的pPager->memDb通过一些简单的C端映射指针.