Aar*_*son 7 sql json redis presto
我试图根据(有限的)presto-redis文档在我的本地机器上设置presto和redis的工作示例.
当使用redis.key-prefix-schema-table=trueredis键并使用前缀dev:simple_table:(按照presto redis连接器页面的指示)时,所有键列都为空,内部列为_key_corrupttrue.正确解析所有值列.
使用redis.key-prefix-schema-table=false和使用纯JSON密钥和值字符串时,两者都按预期工作.(注意:我已尝试过并且未能使用csv dataFormat)
我有一个使用redis服务器和在本地计算机上运行的presto服务器的示例.
presto的etc/catalog/redis.properties文件:
connector.name=redis
redis.table-names=simple_table
redis.default-schema=dev
redis.nodes=127.0.0.1:6379
redis.table-description-dir=/Users/acarson/var/redis/tables
redis.key-delimiter=:
redis.key-prefix-schema-table=false
redis.hide-internal-columns=false
我有一个表定义文件 /Users/acarson/var/redis/tables/simple_table.json
{
"tableName": "simple_table",
"schemaName": "dev",
"key": {
"dataFormat": "json",
"fields": [
{
"name": "id",
"mapping": "id",
"type": "BIGINT"
},
{
"name": "word",
"mapping": "word",
"type": "VARCHAR"
}
]
},
"value": {
"dataFormat": "json",
"fields": [
{
"name": "name",
"mapping": "name",
"type": "VARCHAR"
},
{
"name": "number",
"mapping": "number",
"type": "BIGINT"
},
{
"name": "boolean",
"mapping": "boolean",
"type": "BOOLEAN"
}
]
}
}
使用redis-cli,我用这个命令插入一个值:
SET '{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'
然后我可以使用presto cli查询数据,结果列显示为预期:
presto:dev> SELECT * FROM simple_table;
id | word | name | number | boolean | _key | _value | _key_length | _value_length | _key_corrupt | _value_corrupt
----+------+------+--------+---------+---------------------------+--------------------------------------------------+-------------+---------------+--------------+----------------
42 | foo | bar | 3 | false | {"id": 42, "word": "foo"} | {"name": "bar", "number": 3, "boolean": "false"} | 25 | 48 | false | false
我将redis.properties值切换redis.key-prefix-schema-table为true,重新启动presto,删除redis-cli中的所有键FLUSHALL,然后使用模式表前缀插入新行:
SET 'dev:simple_table:{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'
从presto cli节目查询_key_corrupt=true并且键值为null,即使值仍然有效.
presto:dev> SELECT * FROM simple_table;
id | word | name | number | boolean | _key | _value | _key_length | _value_length | _key_corrupt | _value_corrupt
------+------+--------+--------+---------+-----------------------------------------------+-----------------------------------------------------+-------------+---------------+--------------+----------------
NULL | NULL | bar | 3 | false | dev:simple_table:{"id": 42, "word": "foo"} | {"name": "bar", "number": 3, "boolean": "false"} | 42 | 48 | true | false
我已经增加了redis和presto的日志级别以获取提示,但没有任何显示任何错误或原因导致密钥损坏的原因.我有一种感觉,我的redis键的语法可能不正确,但我遵循presto redis文档中描述的确切方法,甚至在通过presto-redis源读取时指定键使用"schema:table:*"此标志的格式.
以下是运行查询时调试级别的presto日志:
2016-10-20T17:09:55.048-0700 INFO main com.facebook.presto.server.PrestoServer ======== SERVER STARTED ========
2016-10-20T17:10:24.785-0700 DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is PLANNING
2016-10-20T17:10:24.802-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Considering files: [/Users/acarson/var/redis/tables/simple_table.json]
2016-10-20T17:10:24.849-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Redis table dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}}
2016-10-20T17:10:24.850-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Loaded table definitions: [dev.simple_table]
2016-10-20T17:10:24.850-0700 DEBUG Query-20161021_001024_00000_qx72p-104 com.facebook.presto.redis.RedisTableDescriptionSupplier Found Table definition for dev.simple_table: RedisTableDescription{tableName=simple_table, schemaName=dev, key=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=id, type=bigint, mapping=id, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=word, type=varchar, mapping=word, dataFormat=null, formatHint=null, hidden=false}]}, value=RedisTableFieldGroup{dataFormat=json, name=null, fields=[RedisTableFieldDescription{name=name, type=varchar, mapping=name, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=number, type=bigint, mapping=number, dataFormat=null, formatHint=null, hidden=false}, RedisTableFieldDescription{name=boolean, type=boolean, mapping=boolean, dataFormat=null, formatHint=null, hidden=false}]}}
2016-10-20T17:10:25.020-0700 DEBUG query-execution-0 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is STARTING
2016-10-20T17:10:25.027-0700 DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULING
2016-10-20T17:10:25.064-0700 DEBUG query-execution-1 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is SCHEDULED
2016-10-20T17:10:25.065-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is RUNNING
2016-10-20T17:10:25.119-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is RUNNING
2016-10-20T17:10:25.165-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULING
2016-10-20T17:10:25.174-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is SCHEDULED
2016-10-20T17:10:25.179-0700 DEBUG query-execution-2 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is RUNNING
2016-10-20T17:10:25.278-0700 INFO 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisJedisManager Creating new JedisPool for 127.0.0.1:6379
2016-10-20T17:10:25.313-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisRecordCursor Scanning new Redis keys from cursor 0 . 0 values read so far
2016-10-20T17:10:25.326-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.redis.RedisRecordCursor Read a total of 1 values with 48 bytes.
2016-10-20T17:10:25.330-0700 DEBUG 20161021_001024_00000_qx72p.1.0-0-58 com.facebook.presto.execution.TaskExecutor Split 20161021_001024_00000_qx72p.1.0-0 RedisSplit{connectorId=redis, schemaName=dev, tableName=simple_table, keyDataFormat=json, valueDataFormat=json, keyName=null, start=0, end=-1, nodes=[127.0.0.1:6379]} (start = 1477008625258, wall = 72 ms, cpu = 56 ms, calls = 1) is finished
2016-10-20T17:10:25.350-0700 DEBUG http-worker-77 com.facebook.presto.execution.SqlTask Aborting task 20161021_001024_00000_qx72p.1.0 output 0
2016-10-20T17:10:25.352-0700 DEBUG task-notification-1 com.facebook.presto.execution.TaskStateMachine Task 20161021_001024_00000_qx72p.1.0 is FINISHED
2016-10-20T17:10:25.357-0700 DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.1 is FINISHED
2016-10-20T17:10:25.367-0700 DEBUG 20161021_001024_00000_qx72p.0.0-0-59 com.facebook.presto.execution.TaskExecutor Split 20161021_001024_00000_qx72p.0.0-0 (start = 1477008625257, wall = 110 ms, cpu = 9 ms, calls = 4) is finished
2016-10-20T17:10:25.369-0700 DEBUG http-worker-88 com.facebook.presto.execution.SqlTask Aborting task 20161021_001024_00000_qx72p.0.0 output 0
2016-10-20T17:10:25.372-0700 DEBUG task-notification-0 com.facebook.presto.execution.TaskStateMachine Task 20161021_001024_00000_qx72p.0.0 is FINISHED
2016-10-20T17:10:25.379-0700 DEBUG query-execution-0 com.facebook.presto.execution.StageStateMachine Stage 20161021_001024_00000_qx72p.0 is FINISHED
2016-10-20T17:10:25.380-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHING
2016-10-20T17:10:25.383-0700 DEBUG query-execution-2 com.facebook.presto.execution.QueryStateMachine Query 20161021_001024_00000_qx72p is FINISHED
2016-10-20T17:10:25.420-0700 INFO query-execution-2 com.facebook.presto.event.query.QueryMonitor TIMELINE: Query 20161021_001024_00000_qx72p :: Transaction:[c54dc7fe-8159-434d-b4cc-cb13ad41a5d7] :: elapsed 610ms :: planning 247ms :: scheduling 248ms :: running 0ms :: finishing 363ms :: begin 2016-10-20T17:10:24.773-07:00 :: end 2016-10-20T17:10:25.383-07:00
也许这一行需要转义 : 您是否尝试删除它，因为 : 无论如何都是默认值。
redis.key-delimiter=:
另外你的前缀有一个尾部：应该被省略：
SET 'dev:simple_table{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'
代替
SET 'dev:simple_table:{"id": 42, "word": "foo"}' '{"name": "bar", "number": 3, "boolean": "false"}'