小编Pet*_*e C的帖子

jq Streaming - 过滤嵌套列表并保留全局结构

在一个大型 json 文件中,我想从嵌套列表中删除一些元素,但保留文档的整体结构。

我的示例将其输入为(但真实的输入足够大以要求流式传输)。

{
  "keep_untouched": {
    "keep_this": [
      "this",
      "list"
    ]
  },
  "filter_this":
  [
    {"keep" : "true"},
    {
      "keep": "true",
      "extra": "keeper"
    } ,
    {
      "keep": "false",
      "extra": "non-keeper"
    }
  ]
}
Run Code Online (Sandbox Code Playgroud)

所需的输出仅删除了“filter_this”块的一个元素:

{
  "keep_untouched": {
    "keep_this": [
      "this",
      "list"
    ]
  },
  "filter_this":
  [
    {"keep" : "true"},
    {
      "keep": "true",
      "extra": "keeper"
    } ,
  ]
}
Run Code Online (Sandbox Code Playgroud)

处理此类情况的标准方法似乎是使用“truncate_stream”来重构流对象,然后再以通常的 jq 方式过滤这些对象。具体来说,命令:

jq -nc --stream 'fromstream(1|truncate_stream(inputs))' 
Run Code Online (Sandbox Code Playgroud)

提供对对象流的访问:

{"keep_this":["this","list"]}
[{"keep":"true"},{"keep":"true","extra":"keeper"}, 
 {"keep":"false","extra":"non-keeper"}]
Run Code Online (Sandbox Code Playgroud)

此时很容易过滤所需的对象。但是,这会从其父对象的上下文中剥离结果,这不是我想要的。

查看流式结构:

[["keep_untouched","keep_this",0],"this"]
[["keep_untouched","keep_this",1],"list"]
[["keep_untouched","keep_this",1]]
[["keep_untouched","keep_this"]]
[["filter_this",0,"keep"],"true"]
[["filter_this",0,"keep"]]
[["filter_this",1,"keep"],"true"]
[["filter_this",1,"extra"],"keeper"]
[["filter_this",1,"extra"]]
[["filter_this",2,"keep"],"false"] …
Run Code Online (Sandbox Code Playgroud)

json stream-processing bigdata jq

5
推荐指数
1
解决办法
1794
查看次数

标签 统计

bigdata ×1

jq ×1

json ×1

stream-processing ×1