Fluentd可以将日志发送到Logstash吗？

Question

我整天都在尝试这样做。我想通过fluentd日志记录引擎将日志从Docker发送到FluentD，然后从fluentd将那些日志发送到logstash进行处理。

我一直从logstash收到此错误：

{:timestamp=>"2016-03-09T23:29:19.388000+0000",
 :message=>"An error occurred. Closing connection",
 :client=>"172.18.0.1:57259", :exception=>#<TypeError: can't convert String into Integer>,
 :backtrace=>["org/jruby/RubyTime.java:1073:in `at'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-event-2.2.2-java/lib/logstash/timestamp.rb:27:in `at'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.2-java/lib/logstash/codecs/fluent.rb:41:in `decode'", 
"org/msgpack/jruby/MessagePackLibrary.java:195:in `each'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-fluent-2.0.2-java/lib/logstash/codecs/fluent.rb:40:in `decode'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.2/lib/logstash/inputs/tcp.rb:153:in `handle_socket'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-tcp-3.0.2/lib/logstash/inputs/tcp.rb:143:in `server_connection_thread'"], :level=>:error}

相当基本的logstash配置：

input {
  tcp {
    port => 4000
    codec => "fluent"
  }
}

output {
  stdout {
  }
}

相当基本的流利配置：

<source>
  @type forward
</source>


<match docker.json>
  @type forward
  send_timeout 60s 
  recover_wait 10s 
  heartbeat_type none
  phi_threshold 16
  hard_timeout 60s 

  <server>
    name logstash
    host 172.18.0.2
    port 4000
    weight 60
  </server>
</match>

<match docker.**>
  @type stdout
</match>

有人会认为这行得通，但我已经发现Logstash不会：

1. 使用fluentd的forward_out心跳配置。
   • Logstash不会在与TCP相同的端口上打开UDP端口。
2. 上面的错误。

如果我在Ruby中编写Fluentd消息包消息并手动发送，则上述配置确实有效。关键是我希望Fluentd在本地管理日志并将其发送到外部Logstash服务器，以将消息正确处理为JSON。

Answer 1

我们找到了一种使流利的-> logstash工作的方法。设置time_as_integer true。流利的一面的最小配置是

<source>
  @type http
  @id input_http
  port 8888
</source>

<match **>
  @type forward
  time_as_integer true
  <server>
    host localhost
    port 24114
  </server>
</match>

在https://docs.fluentd.org/v0.12/articles/in_forward#i-got-messagepackunknownexttypeerror-error-why中，它被完全隐藏了。在logstash端，使用最新版本（6.2.4），然后只需配置流畅的编解码器，tcp输入，如下所示：

input {
  tcp {
    codec => fluent
    port => 24114
  }
}

filter {
}

output {
  stdout { codec => rubydebug }
}

用...测试

curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test

如文档中所述。使用该time_as_integer设置，logstash输出将看起来不错。

{
          "port" => 32844,
      "@version" => "1",
          "host" => "localhost",
          "json" => "message",
    "@timestamp" => 2018-04-26T15:14:28.000Z,
          "tags" => [
        [0] "debug.test"
    ]
}

没有它，我得到

[2018-04-26T15:16:00,115][ERROR][logstash.codecs.fluent   ] Fluent parse error, original data now in message field {:error=>#<MessagePack::UnknownExtTypeError: unexpected extension type>, :data=>["fluent.info", "\x92\xD7\u0000Z\xE1\xEC\xF4\u0006$\x96?worker\u0000\xA7message\xD9&fluentd worker is now running worker=0", {"size"=>1, "compressed"=>"text"}]}
{
          "port" => 32972,
      "@version" => "1",
       "message" => [
        [0] "fluent.info",
        [1] "\x92\xD7\u0000Z\xE1\xEC\xF4\u0006$\x96?worker\u0000\xA7message\xD9&fluentd worker is now running worker=0",
        [2] {
                  "size" => 1,
            "compressed" => "text"
        }
    ],
          "host" => "localhost",
    "@timestamp" => 2018-04-26T15:16:00.116Z,
          "tags" => [
        [0] "_fluentparsefailure"
    ]
}