标签: mysql-python

我想从MySQL表中读取一些数字(double,即float64)数据.数据大小约为200k行.

MATLAB参考:

tic;
feature accel off;
conn = database(...);
c=fetch(exec(conn,'select x,y from TABLENAME'));
cell2mat(c.data);
toc

经过的时间约为1秒.

在python中做同样的事情,使用这里找到的几个例子(我已经尝试了所有这些,即使用pandas read_frame,frame_query和__processCursor函数): 如何将SQL查询结果转换为PANDAS数据结构？

参考python代码:

import pyodbc
import pandas.io.sql as psql
import pandas
connection_info = "DRIVER={MySQL ODBC 3.51 \
Driver};SERVER=;DATABASE=;USER=;PASSWORD=;OPTION=3;"
cnxn = pyodbc.connect(connection_info)
cursor = cnxn.cursor()
sql = "select x,y from TABLENAME"
#cursor.execute(sql)
#dataframe = __processCursor(cursor, dataframe=True)
#df = psql.frame_query(sql, cnxn, coerce_float=False)
df = psql.read_frame(sql, cnxn)
cnxn.close()

大约需要6秒钟.Profiler说所有花费的时间都在read_frame中.我想知道是否有人能给我一些提示如何加速至少匹配MATLAB代码.如果在python中完全可以的话.

编辑:

瓶颈似乎在pyodbc库中的cursor.execute(在pymysql库中)或cursor.fetchall()内.最慢的部分是按元素(逐行,逐列)读取返回的MySQL数据元素,并将其转换为先前由同一个库推断的数据类型.

到目前为止,我通过这个非常脏的解决方案设法加快了接近MATLAB的速度:

import pymysql
import numpy

conn = pymysql.connect(host='', port=, user='', passwd='', db='')
cursor = conn.cursor() …

我正在制作一个Python项目,我必须从数据库中搜索和检索数据.
我尝试创建一个类,在其中我声明了连接并进行了查询,这里到目前为止我没有更多.

import MySQLdb
dbc =("localhost","root","1234","users")
class sql:
    db = MySQLdb.connect(dbc[0],dbc[1],dbc[2],dbc[3])
    cursor = db.cursor()

    def query(self,sql):
        sql.cursor.execute(sql)
        return sql.cursor.fetchone()

    def rows(self):
        return sql.cursor.rowcount

sqlI = sql()
print(sqlI.query("SELECT `current_points` FROM `users` WHERE `nick` = 'username';"))

因此,主要问题是变量db并且cursor不能从同一个类的其他def /函数调用.我想得到的是一个很好的查询,我可以在那里查询并检索它的内容.这将总结我的代码,因此我应该这样做.

我有一个关于如何将数据帧保存到本地mysql的问题.

import MySQLdb
import pandas as pd
conn=MySQLdb.connect(host="localhost",user='root',passwd="matt123",db="ada")
df=pd.DataFrame(['A','B'],columns=['new_tablecol'])
df.to_sql(name='new_table',con=conn,if_exists='append')

输入此代码后,它说

pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': not all arguments converted during string formatting

我对此很困惑.我可以查询和创建表.但是我无法保存这个数据帧.

美好的一天。希望你一切都好。有人可以帮我解决这个问题吗？

我是MySQL环境的新手。我正在尝试远程连接到MySQL数据库。我使用以下python代码，并收到此错误。

Print(e) = "cryptography is required for sha256_password or 
             caching_sha2_password"

而且不知道如何解决错误。

import pymysql as db

HOST = "XXXXX.XXX.XX”
PORT = XXXX
USER = "my_username"
PASSWORD = "my_password”
DB = "db_name"

try:
    connection = db.Connection(host=HOST, port=PORT,user=USER,                 
    passwd=PASSWORD, db=DB)

    dbhandler = connection.cursor()
    dbhandler.execute("SELECT * from table_name")
    result = dbhandler.fetchall()
    for item in result:
        print (DB)
 except Exception as e:
    print(e)

finally:
    connection.close()

我尝试在 mac 上的虚拟环境中安装 mysqlclient ，使用以下命令：

pip3 install mysqlclient

我收到以下错误：

Collecting mysqlclient
  Using cached https://files.pythonhosted.org/packages/a5/e1/e5f2b231c05dc51d9d87fa5066f90d1405345c54b14b0b11a1c859020f21/mysqlclient-2.0.1.tar.gz
Installing collected packages: mysqlclient
  Running setup.py install for mysqlclient ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/josephestes/Development/apps/vmstracker/tutorial-env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/9_/hr4rf2_914q_g933hy7z4n5h0000gn/T/pip-install-29km7sh3/mysqlclient/setup.py'"'"'; __file__='"'"'/private/var/folders/9_/hr4rf2_914q_g933hy7z4n5h0000gn/T/pip-install-29km7sh3/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/9_/hr4rf2_914q_g933hy7z4n5h0000gn/T/pip-record-0xkt41xv/install-record.txt --single-version-externally-managed --compile --install-headers /Users/josephestes/Development/apps/vmstracker/tutorial-env/include/site/python3.8/mysqlclient
         cwd: /private/var/folders/9_/hr4rf2_914q_g933hy7z4n5h0000gn/T/pip-install-29km7sh3/mysqlclient/
    Complete output (118 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14.6-x86_64-3.8
    creating build/lib.macosx-10.14.6-x86_64-3.8/MySQLdb
    copying MySQLdb/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/MySQLdb
    copying …

我正在尝试在运行CentOS 5.5和python 2.7的机器上安装MySQL-python.这台机器没有运行mysql服务器,这个盒子将使用的mysql实例托管在一个单独的服务器上.我有一个有效的mysql客户端.在尝试时sudo pip install MySQL-python,我得到一个错误EnvironmentError: mysql_config not found,据我所知/etc/my.cnf,这是一个只引用的命令,它也不存在.在我进行一些疯狂的追逐创建虚假的my.cnf文件之前,是否有一种简单的方法来安装MySQL-python？

我想将我的django应用程序从sqlite移植到mysql.

但是当我尝试安装mysql-python时,它给了我这个错误:错误:命令'/ usr/bin/clang'失败,退出状态为1

我环顾四周寻找线索并尝试了这个似乎适用于大多数人的解决方案:

sudo su export CFLAGS = -Qunused-arguments export CPPFLAGS = -Qunused-arguments pip install MySQL-python

完整的错误日志在这里:

pip install MySQL-python
Downloading/unpacking MySQL-python
  Running setup.py egg_info for package MySQL-python

Installing collected packages: MySQL-python
  Running setup.py install for MySQL-python
    building '_mysql' extension
    /usr/bin/clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Qunused-arguments -Qunused-arguments -Dversion_info=(1,2,5,'final',1) -D__version__=1.2.5 -I/usr/local/mysql/include -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c _mysql.c -o build/temp.macosx-10.6-intel-2.7/_mysql.o -Os -g -fno-strict-aliasing -arch x86_64
    xcrun: error: active developer path ("/Users/deeptichopra/Desktop/Applications and Softwares/Xcode.app/Contents/Developer") does not exist, use xcode-select to change
    error: command …

我想指定在执行pandas.read_sql时返回的dtypes.特别是我有兴趣保存内存并将浮点值返回为np.float32而不是np.float64.我知道我之后可以使用astype(np.float32)进行转换,但这并不能解决初始查询中大内存需求的问题.在我的实际代码中,我将提取8400万行,而不是这里显示的5行.pandas.read_csv允许将dtypes指定为dict,但我认为使用read_sql无法做到这一点.

我正在使用MySQLdb和Python 2.7.

顺便说一下,read_sql在运行时(大约2x)似乎比最终的DataFrame存储需要更多的内存.

In [70]: df=pd.read_sql('select ARP, ACP from train where seq < 5', connection)

In [71]: df
Out[71]: 
   ARP      ACP
0  1.17915  1.42595
1  1.10578  1.21369
2  1.35629  1.12693
3  1.56740  1.61847
4  1.28060  1.05935


In [72]: df.dtypes
Out[72]: 
ARP    float64
ACP    float64
dtype: object

我想将MySQL数据库连接到我的django项目,但它抛出一个错误:

"django.core.exceptions.ImproperlyConfigured:请求设置USE_I18N,但未配置设置.您必须先定义环境变量DJANGO_SETTINGS_MODULE或在访问设置之前调用settings.configure()."

跟踪:

 (myenv) LIBINGLADWINs-MacBook-Air:libinrenold$ django-admin dbshell
Traceback (most recent call last):
  File "/Users/libinrenold/Desktop/djangoworks/myenv/bin/django-admin", line 11, in <module>
    sys.exit(execute_from_command_line())
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/core/management/base.py", line 322, in execute
    saved_locale = translation.get_language()
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/utils/translation/__init__.py", line 195, in get_language
    return _trans.get_language()
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/utils/translation/__init__.py", line 59, in __getattr__
    if settings.USE_I18N:
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/Users/libinrenold/Desktop/djangoworks/myenv/lib/python3.6/site-packages/django/conf/__init__.py", line 39, in _setup
    % …

我尝试与 peewee 建立 MySQL 连接，并按照他们网站上的教程进行操作： peewee 快速入门

所以我的代码如下：

from peewee import *

db = MySQLDatabase(
    host='127.0.0.1',
    user='root',
    password='',
    database='db_test'
)

class Person(Model):
    name = CharField()
    birthday = DateField()

    class Meta:
        database = db

class Pet(Model):
    owner = ForeignKeyField(Person, backref='pets')
    name = CharField()
    animal_type = CharField()

    class Meta:
        database = db

db.connect()

db.create_tables([Person, Pet])

db.close()

（我的数据库来自xampp）

但是当我执行此代码时，我收到此错误消息：

peewee.ImproperlyConfigured：MySQL 驱动程序未安装！

我尝试通过安装 此 MySQL 驱动程序来解决此问题。但这完全没有改变。由于我是 python 新手，我不知道如何解决这个问题，如果我只是缺少导入或者我必须使用 pip 安装库？