带有参数化数据类型的Pandas to_sql,如NUMERIC(10,2)

Question

Pandas有一种很好的to_sql方法可以将数据帧写入SQLAlchemy支持的任何RDBMS.

假设我有一个生成的数据帧:

df = pd.DataFrame([-1.04, 0.70, 0.11, -0.43, 1.0], columns=['value'])

如果我尝试将其写入数据库而没有任何特殊行为,我会得到一个双精度的列类型:

df.to_sql('foo_test', an_engine)

如果我想要一个不同的数据类型,我可以指定它(这很好):

df.to_sql('foo_test', an_engine, dtype={'value': sqlalchemy.types.NUMERIC})

但是,如果我想设置列的精度和比例NUMERIC,它会在我脸上爆炸:

df.to_sql('foo_test', an_engine, dtype={'value': sqlalchemy.types.NUMERIC(10,2)})


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-77-dc008463fbfc> in <module>()
      1 df = pd.DataFrame([-1.04, 0.70, 0.11, -0.43, 1.0], columns=['value'])
----> 2 df.to_sql('foo_test', cosd_engine, dtype={'value': sqlalchemy.types.NUMERIC(10,2)})

/Users/igazit/.virtualenvs/myproject/lib/python2.7/site-packages/pandas/core/generic.pyc in to_sql(self, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
    964             self, name, con, flavor=flavor, schema=schema, if_exists=if_exists,
    965             index=index, index_label=index_label, chunksize=chunksize,
--> 966             dtype=dtype)
    967 
    968     def to_pickle(self, path):

/Users/igazit/.virtualenvs/myproject/lib/python2.7/site-packages/pandas/io/sql.pyc in to_sql(frame, name, con, flavor, schema, if_exists, index, index_label, chunksize, dtype)
    536     pandas_sql.to_sql(frame, name, if_exists=if_exists, index=index,
    537                       index_label=index_label, schema=schema,
--> 538                       chunksize=chunksize, dtype=dtype)
    539 
    540 

/Users/igazit/.virtualenvs/myproject/lib/python2.7/site-packages/pandas/io/sql.pyc in to_sql(self, frame, name, if_exists, index, index_label, schema, chunksize, dtype)
   1162             import sqlalchemy.sql.type_api as type_api
   1163             for col, my_type in dtype.items():
-> 1164                 if not issubclass(my_type, type_api.TypeEngine):
   1165                     raise ValueError('The type of %s is not a SQLAlchemy '
   1166                                      'type ' % col)

TypeError: issubclass() arg 1 must be a class

我试图深入研究为什么类型为sqlalchemy.types.NUMERIC1164通过测试,而sqlalchemy.types.NUMERIC(10,2)不是.他们确实有不同的类型(sqlalchemy.sql.visitors.VisitableTypevs sqlalchemy.sql.sqltypes.NUMERIC).

任何线索将不胜感激!

Answer 1

更新:此错误已修复为pandas> = 0.16.0

这是关于最近的一个熊猫错误与0.15.2相同的错误的帖子.

https://github.com/pydata/pandas/issues/9083

协作者建议使用to_sql monkey补丁作为解决它的方法

from pandas.io.sql import SQLTable

def to_sql(self, frame, name, if_exists='fail', index=True,
           index_label=None, schema=None, chunksize=None, dtype=None):
    """
    patched version of https://github.com/pydata/pandas/blob/v0.15.2/pandas/io/sql.py#L1129
    """
    if dtype is not None:
        from sqlalchemy.types import to_instance, TypeEngine
        for col, my_type in dtype.items():
            if not isinstance(to_instance(my_type), TypeEngine):
                raise ValueError('The type of %s is not a SQLAlchemy '
                                 'type ' % col)

    table = SQLTable(name, self, frame=frame, index=index,
                     if_exists=if_exists, index_label=index_label,
                     schema=schema, dtype=dtype)
    table.create()
    table.insert(chunksize)
    # check for potentially case sensitivity issues (GH7815)
    if name not in self.engine.table_names(schema=schema or self.meta.schema):
        warnings.warn("The provided table name '{0}' is not found exactly "
                      "as such in the database after writing the table, "
                      "possibly due to case sensitivity issues. Consider "
                      "using lower case table names.".format(name), UserWarning)

pd.io.sql.SQLDatabase.to_sql = to_sql