Amazon Cloudwatch使用JSON字段记录数据洞察

Question

我试图在其中一个字段中使用包含JSON的数据的Logs Insights,并解析JSON字段

当我使用入门代码将其放入数据时,我的数据如下所示

fields @timestamp, @message
| sort @timestamp desc
| limit 25

如何path在嵌套的JSON中轻松提取变量以对其执行聚合？通过查看一些文档,我认为@message.path可行,但似乎并非如此.有没有人成功解释了Insights中的JSON日志

编辑:我的数据的样本

#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream  i-05d1d61ab853517a0
@message  I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp  1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}

Answer 1

在@pyb洞察力的基础上，我能够parse @message '"path":"*"' as path从中的任何位置提取路径@message。

您可以通过管道传递另一个方法来获得方法，parse @message '"method":"*"' as method而不必担心订购，因为这是第二次全局纯文本搜索@message

如果您@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

使用方法：

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

将导致以下字段：path = '/api/v1/professionals/ID'和method = 'GET'

请注意，这仍然是简单的字符串解析，因此，它没有嵌套键的概念，就像params.format找不到json，但是format只要format您的中没有其他字符串，就使用just will @message。

另请注意，这是针对Insights未在消息中发现JSON的情况。我相信@pyb在这个答案中就是这种情况。使用以下格式未找到我的日志

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"email@example.com","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}

Answer 2

还有正则表达式帮助下的另一个解析

假设你的@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/"}

您可以通过以下方式提取方法

fields @timestamp, @message
| parse @message /\"method\":\"(?<method_type>.*?)\"/

Answer 3

您可以使用该parse命令提取字段。

如果@message是

I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后像这样提取字段：

fields @timestamp, @message
| parse "I, [*T*] INFO -- : {"method":"*"}" as @date, @time, @method
| filter method=GET
| sort @timestamp desc
| limit 20

目前该文档还很轻巧。我可以通过用*正则表达式替换通配符来获得结果，但是解析失败。

Answer 4

CloudWatch Insights日志会自动发现以下日志类型的字段：

Lambda日志

CloudWatch Logs Insights自动发现Lambda日志中的日志字段，但仅针对每个日志事件中的第一个嵌入式JSON片段（注意：重点是我的发现）。如果Lambda日志事件包含多个JSON片段，则可以使用parse命令解析和提取日志字段。有关更多信息，请参阅JSON日志中的字段。

CloudTrail日志

请参阅JSON日志中的字段。

来源： 支持的日志和发现的字段

如果@message是I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后，您可以选择和过滤字段，如下所示：

fields @timestamp, @message, method
| filter method = "GET"
| sort @timestamp desc

它也适用于嵌套字段，即params.format = "json"或results.0.firstName = "Paul"。