使用filebeat,logstash和elasticsearch将json格式日志发送到kibana？

Question

我有这样的日志:

{"logId":"57aaf6c8d32fb","clientIp":"127.0.0.1","time":"03:11:29 pm","uniqueSubId":"57aaf6c98963b","channelName":"JSPC","apiVersion":"v1","modulName":null,"actionName":"apiRequest","typeOfError":"","statusCode":"","message":"In Auth","exception":"In Auth","logType":"Info"}

{"logId":"57aaf6c8d32fb","clientIp":"127.0.0.1","time":"03:11:29 pm","uniqueSubId":"57aaf6c987206","channelName":"JSPC","apiVersion":"v2","modulName":null,"actionName":"performV2","typeOfError":"","statusCode":"","message":"in inbox api v2 5","exception":"in inbox api v2 5","logType":"Info"}

我想推动它们kibana.我使用filebeat将数据发送到logstash,使用以下配置:

filebeat.yml

 ### Logstash as output
logstash:
# The Logstash hosts
hosts: ["localhost:5044"]

# Number of workers per Logstash host.
#worker: 1

现在使用以下配置,我想更改编解码器类型:

input {

     beats {
     port => 5000
     tags => "beats"
     codec => "json_lines"
     #ssl  => true
     #ssl_certificate => "/opt/filebeats/logs.example.com.crt"
     #ssl_key => "/opt/filebeats/logs.example.com.key"
     }


     syslog {
        type => "syslog"
        port => "5514"

    }

}

但是,我仍然以字符串格式获取日志:

"message":"{\"logId \":\"57aaf6c96224b \",\"clientIp \":\"127.0.0.1 \",\"time \":\"03:11:29 pm \",\ "CHANNELNAME \":\ "JSPC \",\ "apiVersion \":空,\ "modulName \":空,\ "actionName \":\ "404 \"\"typeOfError \":\ "例外\" ,"statusCode \":0,\"message \":\"404页面遇到http:\ /\/ localjs.com\/ uploads\/ NonScreenedImages\/ profilePic120\/ 16\/ 29\/15997002iicee52ad041fed55e952d4e4e163d5972ii4c41f8845105429abbd11cc184d0e330.jpeg\"\ "日志类型\":\ "错误\"}",

请帮我解决这个问题.

Answer 1

要解析从Filebeat发送的Logstash中的JSON日志行,您需要使用json过滤器而不是编解码器.这是因为Filebeat将其数据作为JSON发送,并且日志行的内容包含在message字段中.

Logstash配置:

input {
  beats {
    port => 5044
  }   
}   

filter {
  if [tags][json] {
    json {
      source => "message"
    }   
  }   
}   

output {
  stdout { codec => rubydebug { metadata => true } } 
}

Filebeat配置:

filebeat:
  prospectors:
    - paths:
        - my_json.log
      fields_under_root: true
      fields:
        tags: ['json']
output:
  logstash:
    hosts: ['localhost:5044']

在Filebeat配置中,我为事件添加了一个"json"标记,以便可以有条件地将json过滤器应用于数据.

Filebeat 5.0能够在不使用Logstash的情况下解析JSON,但它目前仍然是alpha版本.这篇名为" 使用Filebeat进行结构化日志记录"的博客文章演示了如何使用Filebeat 5.0解析JSON.