logstash和logstash-forwarder之间的字符编码问题

Question

我有以下设置 -

[logstash-forwarder nodes] -> [Amazon's elastic load balancer] -> [logstash nodes]

我使用以下配置文件启动logstash-forwarder -

{
  "network": {
    "servers": ["<Load_balancer_DNS_name>:443"],
    "ssl key": "/etc/pki/private/logstash-forwarder.key",
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
    "timeout": 15
  },

  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "stdin" }
    }
  ]
}

我使用以下设置启动logstash -

input {
  tcp {
    port => "7286"
    codec => plain {
      charset => "UTF-8"
    }
  }
}

output {
  stdout { }
  elasticsearch {
    host => "<cluster_node_ip>"
    protocol => "http"
  }
}

现在我从logstash-forwarder命令行输入一些输入,看它是否在logstash完全可以访问.因此,当我在logstash-forwarder端输入"Hello World"或任何其他纯文本时,我在logstash节点上收到以下内容,而不是原始文本 -

Received an event that has a different character encoding than you configured. {:text=>"1W\\u0000\\u0000\\u0000\\u00011C\\u0000\\u0000\\u0000ox^2ta```\\u0004bV fI\\xCB\\xCCI\\u0005\\xF1uA\\x9C\\x8C\\xFC\\xE2\\u0012 -\\x90Y\\xA0kh\\xA0kha\\xAAkdh\\xACkb\\u0006\\u0014c\\xCBOK+N\\u0005\\xC92\\u001A\\x80\\x94\\xE6d\\xE6\\x81\\xF4\\t\\xBBy\\xFA9\\xFA?\\xB8\\u0006\\x87\\xC4{{:9\\xFA9\\xDA?\\xA4K*\\v@?\\xC5%)\\x99y\\u0000\\u0000\\u0000\\u0000\\xFF\\xFF\\u0001\\u0000\\u0000\\xFF\\xFF\\u001A\\x93\\u0015\\xA2", :expected_charset=>"UTF-8", :level=>:warn}

Answer 1

logstash-forwarder使用唯一的协议与logstash进行通信,名为"lumberjack".

你需要有logstash服务器上的密钥及CRT还有,并使用伐木工人的输入来处理它:

input {
  lumberjack {
    # The port to listen on
    port => 7286

    # The paths to your ssl cert and key
    ssl_certificate => "path/to/logstash-forwarder.crt"
    ssl_key => "path/to/logstash-forwarder.key"

    # Set this to whatever you want.
    type => "somelogs"
  }
}

您所看到的是加密的伐木工人消息.

https://github.com/elasticsearch/logstash-forwarder#use-with-logstash