spark谓词下推是否适用于JDBC？

Question

根据这个

Catalyst应用逻辑优化,例如谓词下推.优化器可以将过滤器谓词下推到数据源中,使物理执行能够跳过不相关的数据.

Spark支持将谓词下推到数据源.此功能是否也适用于JDBC？

(通过检查数据库日志,我可以看到它现在不是默认行为 - 完整查询将传递给数据库,即使它后来受到限制因素限制)

更多细节

使用PostgreSQL 9.4运行Spark 1.5

代码段:

from pyspark import SQLContext, SparkContext, Row, SparkConf
from data_access.data_access_db import REMOTE_CONNECTION

sc = SparkContext()
sqlContext = SQLContext(sc)

url = 'jdbc:postgresql://{host}/{database}?user={user}&password={password}'.format(**REMOTE_CONNECTION)
sql = "dummy"

df = sqlContext.read.jdbc(url=url, table=sql)
df = df.limit(1)
df.show()

SQL跟踪:

< 2015-09-15 07:11:37.718 EDT >LOG:  execute <unnamed>: SET extra_float_digits = 3                                                                                                                      
< 2015-09-15 07:11:37.771 EDT >LOG:  execute <unnamed>: SELECT * FROM dummy WHERE 1=0                                                                                                                   
< 2015-09-15 07:11:37.830 EDT >LOG:  execute <unnamed>: SELECT c.oid, a.attnum, a.attname, c.relname, n.nspname, a.attnotnull OR (t.typtype = 'd' AND t.typnotnull), pg_catalog.pg_get_expr(d.adbin, d.a
drelid) LIKE '%nextval(%' FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON (c.relnamespace = n.oid) JOIN pg_catalog.pg_attribute a ON (c.oid = a.attrelid) JOIN pg_catalog.pg_type t ON (a.a
tttypid = t.oid) LEFT JOIN pg_catalog.pg_attrdef d ON (d.adrelid = a.attrelid AND d.adnum = a.attnum) JOIN (SELECT 15218474 AS oid , 1 AS attnum UNION ALL SELECT 15218474, 3) vals ON (c.oid = vals.oid
 AND a.attnum = vals.attnum)                                                                                                                                                                            
< 2015-09-15 07:11:40.936 EDT >LOG:  execute <unnamed>: SET extra_float_digits = 3                                                                                                                      
< 2015-09-15 07:11:40.964 EDT >LOG:  execute <unnamed>: SELECT "id","name" FROM dummy                                                                                                                   

我希望最后一个选择将包含一个limit 1子句 - 但事实并非如此

Answer 1

Spark DataFrames支持使用JDBC源的谓词下推,但术语谓词以严格的SQL含义使用.这意味着它只涵盖WHERE条款.此外,它似乎仅限于逻辑连接(不IN,OR我害怕)和简单的谓词.

其他所有内容,如限制,计数,排序,组和条件都在Spark方面处理.已经涵盖在SO上的一个警告是,df.count()或者sqlContext.sql("SELECT COUNT(*) FROM df")被转换为SELECT 1 FROM df并且需要使用Spark进行大量数据传输和处理.

这是否意味着失败的原因？不完全是.可以使用任意子查询作为table参数.它比谓词下推更不方便,但其他方面效果很好:

n = ... # Number of rows to take
sql = "(SELECT * FROM dummy LIMIT {0}) AS tmp".format(int(n))
df = sqlContext.read.jdbc(url=url, table=sql)

注意:

一旦Data Source API v2准备就绪,将来可能会改进此行为:

• SPARK-15689
• SPIP:数据源API V2