python-pandas和mysql这样的数据库

Question

Pandas的文档中有许多用于处理以各种格式存储的数据的最佳实践示例.

但是,我无法找到使用MySQL等数据库的任何好例子.

任何人都可以指向我链接或给出一些代码片段,如何使用mysql-python将查询结果转换为Pandas中的数据帧有效吗？

Answer 1

正如Wes所说,一旦你使用DBI兼容库获得数据库连接,io/sql的read_sql就会这样做.我们可以看两个简短的例子,使用MySQLdb和cx_Oracle库连接到Oracle和MySQL并查询它们的数据字典.以下示例为cx_Oracle:

import pandas as pd
import cx_Oracle

ora_conn = cx_Oracle.connect('your_connection_string')
df_ora = pd.read_sql('select * from user_objects', con=ora_conn)    
print 'loaded dataframe from Oracle. # Records: ', len(df_ora)
ora_conn.close()

这是相同的例子MySQLdb:

import MySQLdb
mysql_cn= MySQLdb.connect(host='myhost', 
                port=3306,user='myusername', passwd='mypassword', 
                db='information_schema')
df_mysql = pd.read_sql('select * from VIEWS;', con=mysql_cn)    
print 'loaded dataframe from MySQL. records:', len(df_mysql)
mysql_cn.close()

Answer 2

对于这个问题的最近读者:pandas在他们的14.0版文档中有以下警告:

警告:某些现有函数或函数别名已弃用,将在以后的版本中删除.这包括:tquery,uquery,read_frame,frame_query,write_frame.

和:

警告:不推荐使用DBAPI连接对象时对'mysql'风格的支持.SQLAlchemy引擎(GH6900)将进一步支持MySQL.

这使得许多答案在这里过时了.你应该使用sqlalchemy:

from sqlalchemy import create_engine
import pandas as pd
engine = create_engine('dialect://user:pass@host:port/schema', echo=False)
f = pd.read_sql_query('SELECT * FROM mytable', engine, index_col = 'ID')

Answer 3

为了记录,这是一个使用sqlite数据库的示例:

import pandas as pd
import sqlite3

with sqlite3.connect("whatever.sqlite") as con:
    sql = "SELECT * FROM table_name"
    df = pd.read_sql_query(sql, con)
    print df.shape

Answer 4

我更喜欢用SQLAlchemy创建查询,然后从中创建一个DataFrame.如果您打算一遍又一遍地混合和匹配,SQLAlchemy可以更容易地以Python格式组合SQL条件.

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Table
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from pandas import DataFrame
import datetime

# We are connecting to an existing service
engine = create_engine('dialect://user:pwd@host:port/db', echo=False)
Session = sessionmaker(bind=engine)
session = Session()
Base = declarative_base()

# And we want to query an existing table
tablename = Table('tablename', 
    Base.metadata, 
    autoload=True, 
    autoload_with=engine, 
    schema='ownername')

# These are the "Where" parameters, but I could as easily 
# create joins and limit results
us = tablename.c.country_code.in_(['US','MX'])
dc = tablename.c.locn_name.like('%DC%')
dt = tablename.c.arr_date >= datetime.date.today() # Give me convenience or...

q = session.query(tablename).\
            filter(us & dc & dt) # That's where the magic happens!!!

def querydb(query):
    """
    Function to execute query and return DataFrame.
    """
    df = DataFrame(query.all());
    df.columns = [x['name'] for x in query.column_descriptions]
    return df

querydb(q)

Answer 5

MySQL示例:

import MySQLdb as db
from pandas import DataFrame
from pandas.io.sql import frame_query

database = db.connect('localhost','username','password','database')
data     = frame_query("SELECT * FROM data", database)

Answer 6

同样的语法也适用于使用podbc的Ms SQL服务器.

import pyodbc
import pandas.io.sql as psql

cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER=servername;DATABASE=mydb;UID=username;PWD=password') 
cursor = cnxn.cursor()
sql = ("""select * from mytable""")

df = psql.frame_query(sql, cnxn)
cnxn.close()

Answer 7

这就是你使用psycopg2驱动程序连接到PostgreSQL的方法(如果你使用的是Debian Linux衍生操作系统,请安装"apt-get install python-psycopg2").

import pandas.io.sql as psql
import psycopg2

conn = psycopg2.connect("dbname='datawarehouse' user='user1' host='localhost' password='uberdba'")

q = """select month_idx, sum(payment) from bi_some_table"""

df3 = psql.frame_query(q, conn)