如何将MySQL时间戳(6)读入熊猫？

Question

我有一个MySql表,其时间戳具有微秒分辨率:

+----------------------------+------+
| time                       | seq  | 
+----------------------------+------+
| 2015-06-19 02:17:57.389509 |    0 | 
| 2015-06-19 02:17:57.934171 |   10 |
+----------------------------+------+

我想把它读成一个pandas Dataframe.运用

import pandas as pd
con = get_connection()
result = pd.read_sql("SELECT * FROM MyTable;", con=con)
print result

返回NaT(不是时间):

    time  seq 
0   NaT    0  
1   NaT   10  

如何将其读入时间戳？

Answer 1

通常,要转换时间戳,您可以使用pandas.to_datetime().

>>> import pandas as pd
>>> pd.to_datetime('2015-06-19 02:17:57.389509')
Timestamp('2015-06-19 02:17:57.389509')

从文档中,当从SQL读入时,您可以显式强制将列解析为日期:

pd.read_sql_table('data', engine, parse_dates=['Date'])

或者更明确地,指定格式字符串或要传递给的参数的字典pandas.to_datetime():

pd.read_sql_table('data', engine, parse_dates={'Date': '%Y-%m-%d'})

要么

pd.read_sql_table('data', engine, parse_dates={'Date': {'format': '%Y-%m-%d %H:%M:%S'}})

添加快速概念证明.注意,我正在使用SQLITE.假设您将时间戳存储为字符串:

from sqlalchemy import create_engine, Table, Column, Integer, String, MetaData
import pandas as pd

engine = create_engine('sqlite:///:memory:', echo=True)

datapoints = [{'ts': '2015-06-19 02:17:57.389509', 'seq': 0},
              {'ts':'2015-06-19 02:17:57.934171', 'seq': 10}]
metadata = MetaData()
mydata = Table('mydata', metadata,
    Column('ts', String),
    Column('seq', Integer),
)
metadata.create_all(engine)
ins = mydata.insert()
conn = engine.connect()
conn.execute(ins, datapoints)

foo = pd.read_sql_table('mydata', engine, parse_dates=['ts'])
print(foo)

输出:

                           ts  seq
0  2015-06-19 02:17:57.389509    0
1  2015-06-19 02:17:57.934171   10

或者,如果您将它们存储为日期时间对象,它的工作方式相同(代码差异是我以datetime格式将数据导入数据库):

from datetime import datetime
from sqlalchemy import create_engine, Table, Column, Integer, DateTime, MetaData
import pandas as pd

engine = create_engine('sqlite:///:memory:', echo=True)

datapoints = [{'ts': datetime.strptime('2015-06-19 02:17:57.389509', '%Y-%m-%d %H:%M:%S.%f'), 'seq': 0},
              {'ts':datetime.strptime('2015-06-19 02:17:57.934171', '%Y-%m-%d %H:%M:%S.%f'), 'seq': 10}]
metadata = MetaData()
mydata = Table('mydata', metadata,
    Column('ts', DateTime),
    Column('seq', Integer),
)
metadata.create_all(engine)
ins = mydata.insert()
conn = engine.connect()
conn.execute(ins, datapoints)

foo = pd.read_sql_table('mydata', engine, parse_dates=['ts'])
print(foo)

输出相同:

                          ts  seq
0 2015-06-19 02:17:57.389509    0
1 2015-06-19 02:17:57.934171   10

希望这可以帮助.

编辑为了试图解决@joris的关注,它是真正的sqlite存储所有datetime对象作为字符串,但内置的适配器自动转换这些回datetime对象时取出.扩展第二个例子:

from sqlalchemy.sql import select
s = select([mydata])
res = conn.execute(s)
row = res.fetchone()
print(type(row['ts']))

结果是 <class 'datetime.datetime'>