所以,我们假设我的日志行的一部分看起来像这样:
GET /restAPI/callMethod1/8675309
GET匹配一个http方法,得到提取,余数匹配一个URI,并且也被提取.现在在logstash配置中让我们假设我想做这样的事情......
if [METHOD] == "GET" {
if [URI] (CONTAINS <--Is there a way to do this?) =="restAPI/callMethod1"{
....
有办法做到这一点吗？如果是这样,我该怎么做呢？
谢谢
我正在使用在独立系统（无云或群集）中运行的Logstash-5.6.5（在Windows中）。计划观看一些日志文件并将其发布到本地运行elasticsearch。但是，当检查Logstash的内存使用情况时，如果没有配置监视任何文件的配置，则显示大约600MB内存使用情况。当我进一步添加输入文件管道配置时，它进一步增加了内存（为观看3个日志文件，它总共增加了70MB，但我计划增加多达20个日志）。
1.这是预期的行为吗？
2.有什么方法可以通过logstash减少大量内存使用？
performance memory-management logstash logstash-file logstash-configuration
我一直在尝试使用logstash解析我的python traceback日志.我的日志看起来像这样:
[pid: 26422|app: 0|req: 73/73] 192.168.1.1 () {34 vars in 592 bytes} [Wed Feb 18 13:35:55 2015] GET /data => generated 2538923 bytes in 4078 msecs (HTTP/1.1 200) 2 headers in 85 bytes (1 switches on core 0)
Traceback (most recent call last):
File "/var/www/analytics/parser.py", line 257, in parselogfile
parselogline(basedir, lne)
File "/var/www/analytics/parser.py", line 157, in parselogline
pval = understandpost(parts[3])
File "/var/www/analytics/parser.py", line 98, in understandpost
val = json.loads(dct["events"])
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", …我刚刚开始使用Logstash,并想知道是否有办法在logstash配置文件中指定块注释？当我使用跨越多行的长grok模式进行测试时,它将非常有用.
我一直在尝试配置logstash来读取在我的亚马逊S3存储桶中生成的日志,但是没有成功.以下是详细信息:
input {
s3 {
access_key_id => "MY_ACCESS_KEY_ID"
bucket => "MY_BUCKET"
region => "MY_REGION"
secret_access_key => "MY_SECRET_ACESS_KEY"
prefix => "/"
type => "s3"
add_field => { source => gzfiles }
}
}
filter {
if [type] == "s3" {
csv {
columns => [ "date", "time", "x-edge-location", "sc-bytes", "c-ip", "cs-method", "Host", "cs-uri-stem", "sc-status", "Referer", "User-Agent", "cs-uri-query", "Cookie", "x-edge-result-type", "x-edge-request-id" ]
}
}
if([message] =~ /^#/) {
drop{}
}
}
output {
elasticsearch {
host => "ELASTICSEARCH_URL" protocol … amazon-s3 amazon-web-services logstash logstash-file logstash-configuration
我可以运行logstash:
bin/logstash -f /etc/logstash/conf.d/config.json
但是将logstash作为服务运行
sudo service logstash start
给我以下错误:
... Error: Expected one of #, input, filter, output at line 24, column 1 (byte 528) after "}
不确定我在这里做错了什么？Configtest还说配置文件很好.
bin/logstash -f /etc/logstash/conf.d/config.json --configtest
Configuration OK
任何帮助表示赞赏.
谢谢.
Logstash允许执行任意命令作为管道的输入.这是我的示例管道:
input {
exec {
command => '/usr/bin/ruby -e "puts RUBY_VERSION"'
interval => 10
}
}
output {
stdout { codec => rubydebug }
}
有了这个,我收到以下错误:
/opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:328:in `block in verify_gemfile_dependencies_are_found!': Could not find gem 'logstash-core (= 2.4.0) ruby' in any of the gem sources listed in your Gemfile or installed on this machine. (Bundler::GemNotFound)
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307:in `each'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:307:in `verify_gemfile_dependencies_are_found!'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:199:in `start'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/resolver.rb:182:in `resolve'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:192:in `resolve'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:132:in `specs'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:177:in `specs_for'
from /opt/logstash/vendor/bundle/jruby/1.9/gems/bundler-1.9.10/lib/bundler/definition.rb:166:in `requested_specs' …我使用以下配置启动了logstash:
在logstash.yml里面:
queue.type: persisted
queue.max_bytes: 8gb
queue.checkpoint.writes: 1
配置文件:
input {
beats {
port => "5043"
}
}
filter {
grok {
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "test"
document_type => "tw"
}
}
我有这样的情况.
想象一下,弹性搜索被关闭了
现在想象一下,当关闭elasticsearch时,logstash会收到日志记录事件
现在,如果我打开logstash和elasticsearch,logstash不会发送在步骤2中收到的消息 - 即弹性搜索关闭并且logstash正在接收事件.
如何将此Logstash过滤器更改为不区分大小写？
filter {
if "foo" in [message] {
mutate { add_field => { "Alert_level" => "5" }}
}
}
我对jdbc_driver_library有问题。
ELK_VERSION = 6.4.2
而且我将Docker用于麋鹿。
当我跑步时：
/opt/logstash# bin/logstash -f /etc/logstash/conf.d/mysql.conf
取得错误：
error: com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
驱动路径：
root@xxxxxxx:/etc/logstash/conectors# ls
mysql-connector-java-8.0.12.jar
root@xxxxxxxxxx:/etc/logstash/conectors#
mysql.conf：
input {
jdbc {
jdbc_driver_library => "/etc/logstash/conectors/mysql-connector-java-8.0.12.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/mydb"
jdbc_user => "demouser"
jdbc_password => "demopassword"
statement => "SELECT id,name,city from ads"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => 'test'
document_type => 'tes'
document_id => '%{id}'
hosts => …