为Redshift查询优化大IN条件

Question

我有一个~2TB完全真空的Redshift表,带有一个distkey phash(高基数,数亿个值)和复合排序键(phash, last_seen).

当我做一个查询,如:

SELECT
    DISTINCT ret_field
FROM
    table
WHERE
    phash IN (
        '5c8615fa967576019f846b55f11b6e41',
        '8719c8caa9740bec10f914fc2434ccfd',
        '9b657c9f6bf7c5bbd04b5baf94e61dae'
    )
AND
    last_seen BETWEEN '2015-10-01 00:00:00' AND '2015-10-31 23:59:59'

它很快就会返回.但是,当我将哈希数增加到10以上时,Redshift会将IN条件从一堆OR转换为数组,符合http://docs.aws.amazon.com/redshift/latest/dg/r_in_condition.html#r_in_condition-优化换大的,名单

问题是,当我有几十个phash值时,"优化"查询从不到一秒的响应时间变为超过半小时.换句话说,它停止使用sortkey并进行全表扫描.

知道如何防止这种行为并保留使用sortkeys来保持查询的快速性吗？

这是EXPLAIN<10个哈希和> 10个哈希之间的区别:

少于10(0.4秒):

XN Unique  (cost=0.00..157253450.20 rows=43 width=27)
    ->  XN Seq Scan on table  (cost=0.00..157253393.92 rows=22510 width=27)
                Filter: ((((phash)::text = '394e9a527f93377912cbdcf6789787f1'::text) OR ((phash)::text = '4534f9f8f68cc937f66b50760790c795'::text) OR ((phash)::text = '5c8615fa967576019f846b55f11b6e61'::text) OR ((phash)::text = '5d5743a86b5ff3d60b133c6475e7dce0'::text) OR ((phash)::text = '8719c8caa9740bec10f914fc2434cced'::text) OR ((phash)::text = '9b657c9f6bf7c5bbd04b5baf94e61d9e'::text) OR ((phash)::text = 'd7337d324be519abf6dbfd3612aad0c0'::text) OR ((phash)::text = 'ea43b04ac2f84710dd1f775efcd5ab40'::text)) AND (last_seen >= '2015-10-01 00:00:00'::timestamp without time zone) AND (last_seen <= '2015-10-31 23:59:59'::timestamp without time zone))

超过10(45-60分钟):

XN Unique  (cost=0.00..181985241.25 rows=1717530 width=27)
    ->  XN Seq Scan on table  (cost=0.00..179718164.48 rows=906830708 width=27)
                Filter: ((last_seen >= '2015-10-01 00:00:00'::timestamp without time zone) AND (last_seen <= '2015-10-31 23:59:59'::timestamp without time zone) AND ((phash)::text = ANY ('{33b84c5775b6862df965a0e00478840e,394e9a527f93377912cbdcf6789787f1,3d27b96948b6905ffae503d48d75f3d1,4534f9f8f68cc937f66b50760790c795,5a63cd6686f7c7ed07a614e245da60c2,5c8615fa967576019f846b55f11b6e61,5d5743a86b5ff3d60b133c6475e7dce0,8719c8caa9740bec10f914fc2434cced,9b657c9f6bf7c5bbd04b5baf94e61d9e,d7337d324be519abf6dbfd3612aad0c0,dbf4c743832c72e9c8c3cc3b17bfae5f,ea43b04ac2f84710dd1f775efcd5ab40,fb4b83121cad6d23e6da6c7b14d2724c}'::text[])))

Answer 1

值得一试的是设置sortkeys (last_seen, phash)，last_seen先放。

缓慢的原因可能是因为排序键的前导列phash看起来像随机字符。正如 AWS redshift 开发文档所述，如果将时间戳列用于 where 条件，则时间戳列应作为排序键的前导列。

如果最近的数据查询最频繁，请将时间戳列指定为排序键的前导列。-选择最佳排序键 - Amazon Redshift

使用此排序键顺序，所有列将按last_seen、然后排序phash。（拥有多个排序键列意味着什么？）

需要注意的是，您必须重新创建表才能更改排序键。这将帮助您做到这一点。