使用params从SQL Server读取:pandas(或pyodbc)无法正常运行

Question

我在SQL Server中使用一个查询,需要一个范围来检查一个数字是否在该范围内(例如在下面检查是否DemographicGroupDimID是(1,2或3).在做了一些谷歌搜索之后我找到的唯一方法能够做到这一点如下:

SQL

DECLARE @adults table (Id int)
INSERT INTO @adults VALUES (1), (2), (3)

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM 
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey) 
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]

如果我需要动态地执行此操作,使用不同的范围,则会失败,如下所示:

熊猫查询

from queries import DB_CREDENTIALS
import pyodbc
import pandas as pd

sql_ = """DECLARE @adults table (Id int)
INSERT INTO @adults VALUES ?

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey)
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]"""

with pyodbc.connect(DB_CREDENTIALS) as cnxn:
    df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

错误:

---------------------------------------------------------------------------
DatabaseError                             Traceback (most recent call last)
<ipython-input-5-4b63847d007f> in <module>()
      1 with pyodbc.connect(DB_CREDENTIALS) as cnxn:
----> 2     df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
    497             sql, index_col=index_col, params=params,
    498             coerce_float=coerce_float, parse_dates=parse_dates,
--> 499             chunksize=chunksize)
    500 
    501     try:

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in read_query(self, sql, index_col, coerce_float, params, parse_dates, chunksize)
   1593 
   1594         args = _convert_params(sql, params)
-> 1595         cursor = self.execute(*args)
   1596         columns = [col_desc[0] for col_desc in cursor.description]
   1597 

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in execute(self, *args, **kwargs)
   1570             ex = DatabaseError(
   1571                 "Execution failed on sql '%s': %s" % (args[0], exc))
-> 1572             raise_with_traceback(ex)
   1573 
   1574     @staticmethod

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in execute(self, *args, **kwargs)
   1558                 cur.execute(*args, **kwargs)
   1559             else:
-> 1560                 cur.execute(*args)
   1561             return cur
   1562         except Exception as exc:

DatabaseError: Execution failed on sql 'DECLARE @adults table (Id int)
INSERT INTO @adults VALUES ?

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey)
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]': ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '@P1'. (102) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")

这是因为declare语句需要在select语句本身的范围内吗？我不确定如何pandas处理pyodbc光标对象,所以我不确定这个错误源于何处.

编辑:请注意,我在这个实例中传递的参数(30)只是使用了范围中只有一个数字失败的简单情况.它当然也失败了更复杂的字符串,就像(1), (2), (3)上面的例子一样.

Answer 1

如果在SQL中使用预准备语句,则不能为一个占位符/参数/绑定变量放置多个值!

除此之外,您只能使用占位符/参数/绑定变量来代替文字,您不能将它用于不是文字的SQL语句的一部分.

在你的情况下,你试图放置(,)哪个是SQL的一部分,但不是文字作为参数.

使用参数/预准备语句/绑定变量也可以保护您免受某些SQL注入.

说,尝试更改您的代码如下:

更改

INSERT INTO @adults VALUES ?

至

INSERT INTO @adults VALUES (?)

和

df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

至

df = pd.read_sql(sql=sql_, con=cnxn, params=['30'])

更新:

你可以这样准备你的SQL:

In [9]: vals = [20,30,40]

In [32]: vals
Out[32]: [20, 30, 40]

In [33]: ' (?)' * len(vals)
Out[33]: ' (?) (?) (?)'

然后:

In [14]: sql_ = """DECLARE @adults table (Id int)
   ....: INSERT INTO @adults VALUES {}
   ....:
   ....: SELECT [date],
   ....: """

In [15]: sql_.format(' (?)' * len(vals))
Out[15]: 'DECLARE @adults table (Id int)\nINSERT INTO @adults VALUES (?) (?) (?)\n\nSELECT [date],\n'

注意生成 (?) (?) (?)

最后调用你的SQL:

df = pd.read_sql(sql=sql_.format(' (?)' * len(vals)), con=cnxn, params=vals)