一、设置kafka
参考文档：
配置参考文档： https://www.orchome.com/171
https://blog.csdn.net/difffate/article/details/53570344
https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756
https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756/5
文档配置：
https://blog.csdn.net/qq_41926119/article/details/104510481
kafka官方文档：
1. 生成证书：
keytool -keystore kafka.server.keystore.jks -alias xxx -validity 3000 -genkey -keyalg RSA
openssl req -new -x509 -keyout ca-key -out ca-cert -days 3000 -passout pass:xxx
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias xxx -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3000 -CAcreateserial -passin pass:xxx
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias xx -import -file cert-signed
2.基于jks生成pem证书：
keytool -importkeystore -srckeystore kafka.server.keystore.jks -destkeystore client.p12 -deststoretype PKCS12
openssl pkcs12 -in client.p12 -nokeys -out client.cer.pem
openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key.pem
keytool -exportcert -alias xxx -keystore kafka.client.truststore.jks -rfc -file serverpub.pem
3. 查看pem证书：
keytool -printcert -file certificate.pem
多个pem证书合并：
cat cert1.pem cert2.pem > bundle.pem
4. 查看jks证书:
keytool -list -v -keystore server.truststore.jks
5.生成filebeat证书:（需要把ca.pem 加入到 kafka的truststore.jks中，完成服务端授信）
通过ES生成自签的CA证书：./bin/elasticsearch-certutil ca
生成颁发的证书：./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --name filebeat --pem --out filebeat2.zip
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert
6. kafka的最终配置
listeners=PLAINTEXT://172.28.15.231:9092,SSL://172.28.15.231:9093
ssl.endpoint.identification.algorithm=
ssl.keystore.location=/data/xxx/kafka_ca_cert/kafka.server.keystore.jks
ssl.keystore.password=xxxx
ssl.truststore.location=/data/xxx/kafka_ca_cert/kafka.server.truststore.jks
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.secure.random.implementation=SHA1PRNG
7. 客户端配置：client-ssl.properties
security.protocol=SSL
ssl.truststore.location=/data/xxx/kafka_ca_cert/kafka.client.truststore.jks
ssl.truststore.password=xxxx
ssl.keystore.type=JKS
ssl.truststore.type=JKS
ssl.endpoint.identification.algorithm=
8. kafka 客户端生产：
kafka-console-producer.sh --topic test-log --broker-list xxxx:9093 --producer.config client-ssl.properties
kafka-console-producer.sh --topic test-log --broker-list xxxx:9093 --producer.config client-ssl.properties
9. kafka 客户端消费：
kafka-console-consumer.sh --topic test-log --bootstrap-server xxxx:9093 --consumer.config client-ssl.properties
二、filebeat的一些配置
最终filebeat配置：
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/archived/*/*
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.kafka:
#指定版本号
version: "0.9.0"
enabled: true
hosts:
- 172.28.15.231:9093
topic: "test-log"
version: "2.0.0"
ssl:
enabled: true
certificate: '/data/xxx/kafka_ca_cert/filebeat/filebeat.crt'
key: '/data/xxx/kafka_ca_cert/filebeat/filebeat.key'
certificate_authorities: ['/data/xxx/kafka_ca_cert/serverpub.pem']
supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]
verification_mode: full
endpoint:
identification:
algorithm:
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
filebeat 的一些配置的官方文档：
https://www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html
https://www.elastic.co/guide/en/beats/filebeat/master/configuration-ssl.html
https://blog.csdn.net/qq_41926119/article/details/104510481
EFK接入kafka消息队列:
https://www.ucloud.cn/yun/34441.html
需要注意的是，filebeat配置的是pem证书，kafka和logstash的kafka-input插件用的是jks证书~~~因此，证书生成工具最好需要能够同时生成这两种证书。
常见错误集锦：
https://discuss.elastic.co/t/tls-for-filebeat-kafka-output/58756/11
其他配置：
Filebeat与Logstash配置SSL加密通信：
http://www.manongjc.com/detail/14-jogxwlmgmlwauxs.html
https://www.cnblogs.com/sanduzxcvbnm/p/12055038.html
filebeat常用配置项：https://www.elastic.co/guide/en/beats/filebeat/7.x/kafka-output.html
SSL配置项： https://www.elastic.co/guide/en/beats/filebeat/7.16/configuration-ssl.html#client-verification-mode
