使用jq从ElasticSearch结果中的_source文档中提取数据

Question

我正在阅读这个StackOverFlow讨论，将JSON转换为CSV，这似乎很棒，但是我无法使用基本的jq来工作。.我不确定我在做什么错。我已经尝试了基本的操作，但是我无法破解错误。这是我在Shell脚本中的ES查询

curl -XGET 'http://es-1:9200/data_latest/customer/_search?pretty' -H 'Content-Type: application/json' -d'
{
"_source": ["customer_app_version", "customer_num_apps", "customer_name","app_disk_size_bytes","app_memory_capacity_bytes"],
    "query": {
        "bool": {
            "must": [{
                "term": {
                    "is_app_customer": {
                        "value": "true"
                    }
                }
            }]
        }
    },
    "aggs": {
        "Customer_UUID": {
            "terms": {
                "field": "customer_uuid",
                "size": 100
            }
        }
    }
}

“ Shell脚本输出

{
  "took": 8,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 6171,
    "max_score": 1.8510876,
    "hits": [
      {
        "_index": "data_latest_v1",
        "_type": "customer",
        "_id": "0003245-4844-9015-1z2e-d4ae5234rd56",
        "_score": 1.8510876,
        "_source": {
          "customer_app_version": "el7.20150513",
          "customer_num_apps": 3,
          "app_memory_capacity_bytes": 405248409600,
          "customer_name": "Timbuktu Inc",
          "app_disk_size_bytes": 25117047875604
        }
      },
      {
        "_index": "data_latest_v1",
        "_type": "customer",
        "_id": "0003245-4844-9015-1z2e-d4ae5234rd56",
        "_score": 1.8510876,
        "_source": {
          "customer_app_version": "el4.20150513",
          "customer_num_apps": 34,
          "app_memory_capacity_bytes": 58923439600,
          "customer_name": "Bunnies Inc",
          "app_disk_size_bytes": 36517984275604
        }
      }
    ]
  }
}

（被截断，但是上面的子集在语法上有效）

1. 如何在Shell脚本中使用jq将_source字段中的键和值（没有其他内容）输出为CSV？我知道我要问的是其他讨论中描述的问题，但我尝试了却无法理解

例如，我在添加（jq -r'。“客户名称”'

并尝试了

| jq -r'.customer_name'

对于两者，我都会得到这样的输出。

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
103 13566  100 13566    0   346   507k  13248 --:--:-- --:--:-- --:--:--  537k
null

我究竟做错了什么？我需要做什么？如果有人可以在这里指导我，那将超级有帮助。

Answer 1

要在jq查询中描述如何在文档中导航到要提取的数据，可能类似于以下内容：

jq -r '.hits.hits[]._source.customer_name'

在这种情况下，输出为：

jq -r '.hits.hits[]._source.customer_name'

要生成键/值CSV，可以使用：

jq -r '.hits.hits[]._source | to_entries | .[] | [.key, .value] | @csv'

...输出：

"customer_app_version","el7.20150513"
"customer_num_apps",3
"app_memory_capacity_bytes",405248409600
"customer_name","Timbuktu Inc"
"app_disk_size_bytes",25117047875604
"customer_app_version","el4.20150513"
"customer_num_apps",34
"app_memory_capacity_bytes",58923439600
"customer_name","Bunnies Inc"
"app_disk_size_bytes",36517984275604

如果您希望客户名称是其自己的列，则可能是：

jq -r '.hits.hits[]._source | .customer_name as $name | del(.customer_name) | to_entries | .[] | [$name, .key, .value] | @csv'

...输出：

"Timbuktu Inc","customer_app_version","el7.20150513"
"Timbuktu Inc","customer_num_apps",3
"Timbuktu Inc","app_memory_capacity_bytes",405248409600
"Timbuktu Inc","app_disk_size_bytes",25117047875604
"Bunnies Inc","customer_app_version","el4.20150513"
"Bunnies Inc","customer_num_apps",34
"Bunnies Inc","app_memory_capacity_bytes",58923439600
"Bunnies Inc","app_disk_size_bytes",36517984275604

如果您愿意对列名进行硬编码，请考虑：

jq -r '.hits.hits[]._source | [.customer_name, .customer_app_version, .customer_num_apps, .app_memory_capacity_bytes, .app_disk_size_bytes] | @csv'

输出：

"Timbuktu Inc","el7.20150513",3,405248409600,25117047875604
"Bunnies Inc","el4.20150513",34,58923439600,36517984275604