如何将Stackdriver日志导入BigQuery

Question

有没有办法将应用引擎中的日志加载到Google Cloud Platform上的BigQuery？

我正在尝试使用联合查询来加载云存储中的Stackdriver日志文件.但是BigQuery无法加载Stackdriver编写的一些字段名称.

日志文件是换行符分隔的JSON,其记录看起来像

{  
   "insertId":"j594356785jpk",
   "labels":{  
      "appengine.googleapis.com/instance_name":"aef-my-instance-20180204t220251-x59f",
      "compute.googleapis.com/resource_id":"99999999999999999",
      "compute.googleapis.com/resource_name":"c3453465db",
      "compute.googleapis.com/zone":"us-central1-f"
   },
   "logName":"projects/my-project/logs/appengine.googleapis.com%2Fstdout",
   "receiveTimestamp":"2018-02-08T02:59:59.972739505Z",
   "resource":{  
      "labels":{  
         "module_id":"my-instance",
         "project_id":"my-project",
         "version_id":"20180204t220251"
      },
      "type":"gae_app"
   },
   "textPayload":"{\"json\":\"blob\"}\n",
   "timestamp":"2018-02-08T02:59:54Z"
}

但BigQuery在此输入上返回错误: query: Invalid field name "compute.googleapis.com/zone". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long.

有没有办法将这种日志摄入BigQuery？

我特别感兴趣的是只提取textPayload字段.

Answer 1

您可以将Stackdriver日志导出到BigQuery中

在Google Cloud Platform控制台中,Stackdriver日志记录 >> 导出 >> 创建导出.创建一个过滤器resource.type="gae_app"并填写右边的Sink Name,然后选择BigQuery作为Sink Service,然后选择要从Sink Destination导出到的数据集.https://console.cloud.google.com/logs/exports

参考:https://cloud.google.com/logging/docs/export/configure_export_v2#dest-create