前言
logstash是ELK日志系统中的一部分，主要承担将收集完成日志进行过滤，并且输出到es的职责。
Logstash的事件（logstash将数据流中等每一条数据称之为一个event）处理流水线有三个主要角色完成：inputs –> filters –> outputs：
inpust：必须，负责产生事件（Inputs generate events），常用：File、syslog、redis、beats（如：Filebeats）
filters：可选，负责数据处理与转换（filters modify them），常用：grok、mutate、drop、clone、geoip
outpus：必须，负责数据输出（outputs ship them elsewhere），常用：elasticsearch、file、graphite、statsd。
需求：需要把数据库oracle 里的数据同步到es中。所以考虑要用到logstasg里的插件logstash-input-jdbc 所以先安装logstash
1. 下载 （linux版本）：（版本 7.3.2）
官网地址：https://www.elastic.co/downloads/logstash
本人在网盘中下载比较快，国外那个太慢了，一直超时。 地址 https://pan.baidu.com/s/1CYsQvfhLoRAGUA6-hRRuqw 提取码：ats6
2:解压安装
命令： tar -zxvf logstash-7.3.2.tar.gz
启动 进去到bin目录下 执行
./logstash -e 'input { stdin { } } output { stdout {} }'
控制台输出：
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/logstash-7.3.2/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /home/logstash-7.3.2/logs which is now configured via log4j2.properties
[2020-03-01T07:10:08,891][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/home/logstash-7.3.2/data/queue"}
[2020-03-01T07:10:08,922][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/home/logstash-7.3.2/data/dead_letter_queue"}
[2020-03-01T07:10:09,491][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-03-01T07:10:09,513][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.2"}
[2020-03-01T07:10:09,562][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"cdb36cc9-faca-451d-8d3d-b9d447141467", :path=>"/home/logstash-7.3.2/data/uuid"}
[2020-03-01T07:10:11,071][INFO ][org.reflections.Reflections] Reflections took 79 ms to scan 1 urls, producing 19 keys and 39 values
[2020-03-01T07:10:12,198][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-03-01T07:10:12,206][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x7e41ce73 run>"}
[2020-03-01T07:10:28,362][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2020-03-01T07:10:28,476][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-03-01T07:10:28,811][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
表示启动成功.
测试 输入数据
helloword
{
"@version" => "1",
"host" => "0.0.0.0",
"message" => "helloword",
"@timestamp" => 2020-02-29T23:11:07.342Z
}
