Elasticsearch 是一个实时的分布式搜索分析引擎，它能让你以前所未有的速度和规模，去探索你的数据
Elastic 家族
| 开源 | 商业 |
|---|---|
| Kibana(可视化) | X-Pack (安全/告警/监控/图查询) |
| Elasticsearch(存储/计算) | |
| Logstash/Beat (数据抓取) |
BEATS - 轻量级数据采集器
ElasticSearch 的目录结构
| 目录 | 配置文件 | 描述 |
|---|---|---|
| bin | 脚本文件，包括启动elasticsearch，安装插件，运行统计数据等 | |
| config | elasticsearch.yml | 集群配置文件，user，role based 相关配置 |
| JDK | java 运行环境 | |
| data | path.data | 数据文件 |
| lib | java 类库 | |
| logs | path.log | 日志文件 |
| modules | 包含所有ES模块 | |
| plugins | 包含所有已经安装的插件 |
开始安装
安装环境 Centos7.5
1、安装 elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.2.0-linux-x86_64.tar.gz
tar -xf elasticsearch-7.2.0-linux-x86_64.tar.gz
cd elasticsearch-7.2.0
./bin/elasticsearch # 启动
启动成功后访问本地的 9200 端口，可以看到
{
"name": "localhost.localdomain",
"cluster_name": "elasticsearch",
"cluster_uuid": "4c-wiX4DSWCtNUZPQxTy5g",
"version": {
"number": "7.2.0",
"build_flavor": "default",
"build_type": "tar",
"build_hash": "508c38a",
"build_date": "2019-06-20T15:54:18.811730Z",
"build_snapshot": false,
"lucene_version": "8.0.0",
"minimum_wire_compatibility_version": "6.8.0",
"minimum_index_compatibility_version": "6.0.0-beta1"
},
"tagline": "You Know, for Search"
}
如果有安装的错误，参考：
seccomp unavailable 错误
解决方法：elasticsearch.yml 配置
bootstrap.memory_lock: false
bootstrap.system_call_filter: falsemax file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
解决方法：修改 /etc/security/limits.conf，配置：
hard nofile 80000
soft nofile 80000max virtual memory areas vm.max_map_count [65530] is too low
解决方法：修改 /etc/sysctl.conf，添加 ：
vm.max_map_count = 262144
然后 sysctl -p 生效the default discovery settings are unsuitable...., last least one of [....] must be configured
解决方法：elasticsearch.yml 开启配置：
node.name: node-1
cluster.initial_master_nodes: ["node-1"]
2、安装插件
Elasticsearch的 ICU 分析器插件 使用 国际化组件 Unicode (ICU) 函数库（详情查看 site.project.org ）提供丰富的处理 Unicode 工具。 这些包含对处理亚洲语言特别有用的
icu_分词器，还有大量对除英语外其他语言进行正确匹配和排序所必须的分词过滤器。
./bin/elasticsearch-plugin install analysis-icu
运行多个Elasticsearch 实例
./bin/elasticsearch -E node.name=node0 -E cluster.name=myes -E path.data=node0_data -d
./bin/elasticsearch -E node.name=node1 -E cluster.name=myes -E path.data=node1_data -d
./bin/elasticsearch -E node.name=node2 -E cluster.name=myes -E path.data=node2_data -d
查看运行的节点
curl -XGET http://127.0.0.1:9200/_cat/nodes
127.0.0.1 15 54 3 1.88 0.67 0.24 mdi * node0
127.0.0.1 12 54 3 1.88 0.67 0.24 mdi - node2
127.0.0.1 12 54 3 1.88 0.67 0.24 mdi - node1
安装Kibana
1. 安装kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.2.0-linux-x86_64.tar.gz
tar -xf kibana-7.2.0-linux-x86_64.tar.gz
cd kibana-7.2.0-linux-x86_64
2.启动kibana
# 将kibana改成中文
vim config/kibana.yml
i18n.locale: "zh-CN" ## 最后一行
./bin/kibana
访问本地的5601端口
查看样例，这是一个电商网站的订单。
// 查看插件
bin/kiban-plugin list
打开 dev tools。
这个console是直接对接es的，所以可以在这里直接使用查询语句。
安装Logstash
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.2.0.tar.gz
运行。
准备 配置文件 logstash.conf
运行 bin/logstash -f logstash.conf
测试
下在测试样本集
wget http://files.grouplens.org/datasets/movielens/ml-20m.zip
unzip ml-20m.zip
配置一份配置文件 log.conf
input {
file {
path => "/home/work/logs/ml-20m/movies.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["id","content","genre"]
}
mutate {
split => { "genre" => "|" }
remove_field => ["path", "host","@timestamp","message"]
}
mutate {
split => ["content", "("]
add_field => { "title" => "%{[content][0]}"}
add_field => { "year" => "%{[content][1]}"}
}
# mutate {
# gsub => [
#
# "year", "\\)", ""
# ]
# }
mutate {
convert => {
"year" => "integer"
}
strip => ["title"]
remove_field => ["path", "host","@timestamp","message","content"]
}
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "movies"
document_id => "%{id}"
}
stdout {}
}
配置 JAVA 运行环境 并开始运行
logstash -f log.conf
# 运行后，数据将写入到es中
至此为止，一个elk 搭建完成
内核参数修改（参考）
32C + 128G
grep "* - nofile 512000" /etc/security/limits.conf || echo "* - nofile 512000" >> /etc/security/limits.conf #修改文件描述符数量
grep "work - nproc unlimited" /etc/security/limits.conf || echo "elasticsearch - nproc unlimited" >> /etc/security/limits.conf #修改最大打开进程数数量
grep "* soft memlock unlimited" /etc/security/limits.conf || echo "* soft memlock unlimited" >> /etc/security/limits.conf #配合es mem lock，centos6无须添加
grep "* hard memlock unlimited" /etc/security/limits.conf || echo "* hard memlock unlimited" >> /etc/security/limits.conf #配合es mem lock，centos6无须添加
grep "fs.file-max = 1024000" /etc/sysctl.conf || echo "fs.file-max = 1024000" >> /etc/sysctl.conf #修改系统文件描述符
grep "vm.max_map_count = 262144" /etc/sysctl.conf || echo "vm.max_map_count = 262144" >> /etc/sysctl.conf #修改程序最大管理的vm
grep "vm.min_free_kbytes = 2097152" /etc/sysctl.conf || echo "vm.min_free_kbytes = 2097152" >> /etc/sysctl.conf
grep "vm.zone_reclaim_mode = 0" /etc/sysctl.conf || echo "vm.zone_reclaim_mode = 0" >> /etc/sysctl.conf
sysctl -p
swapoff -a #关闭虚拟内存
