加载到Google Big Query时出现JSON格式错误

Question

我正在尝试使用内置的数据流模板从PUBSUB加载Big Query中的以下数据:

{
    "current_speed": "19.09",
    "_east": "-87.654561",
    "_last_updt": "2018-07-17 15:50:54.0",
    "_region_id": "1",
    "_north": "42.026444",
    "_south": "41.997946",
    "region": "Rogers Park - West Ridge",
    "_west": "-87.709645",
    "_description": "North of Devon. Kedzie to Lake Shore"
}

但我一直收到这个错误:

"读取数据时出错,错误消息:无法解析JSON:字符串意外结束;字符串意外结束;预期键"

我实际上需要加载更大的数据集,如下所示:

 [{
    "current_speed": "19.09",
    "_east": "-87.654561",
    "_last_updt": "2018-07-17 15:50:54.0",
    "_region_id": "1",
    "_north": "42.026444",
    "_south": "41.997946",
    "region": "Rogers Park - West Ridge",
    "_west": "-87.709645",
    "_description": "North of Devon. Kedzie to Lake Shore"
}, {
    "current_speed": "25.23",
    "_east": "-87.747456",
    "_last_updt": "2018-07-17 15:50:54.0",
    "_region_id": "2",
    "_north": "42.0190998",
    "_south": "41.960669",
    "region": "Far North West",
    "_west": "-87.84621",
    "_description": "North of Montrose. East River to Cicero"
}

]

但是我得到了这个错误:

读取数据时出错,错误消息:无法解析JSON:启动新阵列时找不到对象.BeginArray返回false; 解析器在字符串结束之前终止

我在这做错了什么？

Answer 1

是的,BigQuery只接受换行符分隔的JSON,这意味着每行一个完整的JSON对象.在将对象合并到一行之前,BigQuery读取"{",它是对象的开头,并期望读取一个键,但行结束,因此您会看到错误消息"expected key".

对于多个JSON对象,只需将它们放在每一行中.不要将它们包含在数组中.BigQuery期望每一行都以一个对象"{"开头.如果您将"["作为第一个字符,您将看到第二条错误消息,这意味着BigQuery读取数组但不读取对象内部.

Answer 2

要将JSON转换为新行分隔的JSON（BigQuery提取的格式），可以使用jq：

$ cat a.json 
[{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
},
{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
},
{
    "key01": "value01",
    "key02": "value02",
    "keyN": "valueN"
}
]


$ cat a.json | jq -c '.[]'
{"key01":"value01","key02":"value02","keyN":"valueN"}
{"key01":"value01","key02":"value02","keyN":"valueN"}
{"key01":"value01","key02":"value02","keyN":"valueN"}

（请参阅/sf/answers/3591075281/）