我有logstash在主机上查看目录.每当它看到与我在logstash配置中指定的路径匹配的日志时,它就会开始将数据导入到我的elasticsearch集群中.logstash是否有办法在完成消耗之后删除日志？
我想写脚本来删除logstash已经完成的日志,但我怎么知道他做了哪些日志？
也许你们之前做过这个或知道如何实现这个？
我正在使用带有logstash的csv格式读取检查点日志文件,并且某些字段具有空值.
我想删除所有具有空值的字段.
我无法确切地预测哪些字段(键)将具有空值,因为我在csv文件中有150列,我不想检查它们中的每一个.
是否可以在logstash中执行动态过滤器,删除任何具有空值的字段？
我的logstash配置文件看起来像这样:
input {
stdin { tags => "checkpoint" }
file {
type => "file-input"
path => "D:\Browser Downloads\logstash\logstash-1.4.2\bin\checkpoint.csv"
sincedb_path => "D:\Browser Downloads\logstash\logstash-1.4.2\bin\sincedb-access2"
start_position => "beginning"
tags => ["checkpoint","offline"]
}
}
filter {
if "checkpoint" in [tags] {
csv {
columns => ["num","date","time","orig","type","action","alert","i/f_name","i/f_dir","product","Internal_CA:","serial_num:","dn:","sys_message:","inzone","outzone","rule","rule_uid","rule_name","service_id","src","dst","proto","service","s_port","dynamic object","change type","message_info","StormAgentName","StormAgentAction","TCP packet out of state","tcp_flags","xlatesrc","xlatedst","NAT_rulenum","NAT_addtnl_rulenum","xlatedport","xlatesport","fw_message","ICMP","ICMP Type","ICMP Code","DCE-RPC Interface UUID","rpc_prog","log_sys_message","scheme:","Validation log:","Reason:","Serial num:","Instruction:","fw_subproduct","vpn_feature_name","srckeyid","dstkeyid","user","methods:","peer gateway","IKE:","CookieI","CookieR","msgid","IKE notification:","Certificate DN:","IKE IDs:","partner","community","Session:","L2TP:","PPP:","MAC:","OM:","om_method:","assigned_IP:","machine:","reject_category","message:","VPN internal source IP","start_time","connection_uid","encryption failure:","vpn_user","Log ID","message","old IP","old port","new IP","new port","elapsed","connectivity_state","ctrl_category","description","description ","severity","auth_status","identity_src","snid","src_user_name","endpoint_ip","src_machine_name","src_user_group","src_machine_group","auth_method","identity_type","Authentication trial","roles","dst_user_name","dst_machine_name","spi","encryption fail reason:","information","error_description","domain_name","termination_reason","duration"]
# remove_field => [ any …