微服务异常太乱，我们如何分类？

Elastic APM 是基于 Elastic Stack 构建的应用性能监控系统。通过 Elastic APM 可以监控应用程序，收集有关请求的响应时间、数据库查询、高速缓存调用、外部 HTTP 请求等的详细性能信息，这样可以更快地查明并修复性能问题。

Elastic APM 还会自动收集未处理的错误和异常，错误主要基于堆栈跟踪进行分组，因此可以识别出现的新错误，并密切关注特定错误发生的次数。

Elastic APM

kibana

Kibana 是开源的分析和可视化平台，旨在与 Elasticsearch 协同工作，可以通过 Kibana 搜索、查看 Elasticsearch 中存储的数据，此处用于可视化 Elasticsearch 中存储的 APM 数据

下载

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-linux-x86_64.tar.gz

解压

tar -zxvf kibana-6.5.4-linux-x86_64.tar.gz 
cd kibana-6.5.4-linux-x86_64/

配置

vi config/kibana.yml

[root@JD kibana-6.5.4-linux-x86_64]# vi config/kibana.yml 
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://47.99.88.28:9200"

启动

nohup ./bin/kibana &

访问Kibana

image

APM监控

image

搭建apm-server

APM Server 是用 Go 编写的开源应用程序，通常运行在专用服务器上，默认监听端口 8200 ，并通过 JSON HTTP API 从 agent 接收数据，然后根据该数据创建文档并将其存储在 Elasticsearch 中。

wget https://artifacts.elastic.co/downloads/apm-server/apm-server-6.5.4-linux-x86_64.tar.gz

配置文件

[root@demo03 apm-server-6.5.4-linux-x86_64]# vi apm-server.yml 
apm-server:
  # Defines the host and port the server is listening on.  use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "0.0.0.0:8200"
  run:
    enabled: true
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["47.99.88.28:9200"]

  indices:
    - index: "apm-%{[beat.version]}-sourcemap"
      when.contains:
        processor.event: "sourcemap"

    - index: "apm-%{[beat.version]}-error-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "error"

    - index: "apm-%{[beat.version]}-transaction-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "transaction"

    - index: "apm-%{[beat.version]}-span-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "span"

    - index: "apm-%{[beat.version]}-metric-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "metric"

    - index: "apm-%{[beat.version]}-onboarding-%{+yyyy.MM.dd}"
      when.contains:
        processor.event: "onboarding"

启动

nohup ./apm-server -e  >&/dev/null &

下载

cd app/agent
wget https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/1.4.0/elastic-apm-agent-1.4.0.jar

应用埋点APM agent

APM agent 是使用与服务相同的语言编写的开源库，可以像安装其他库一样将它们安装到服务中，agent 将检测服务的代码并在运行时收集性能数据和错误，这些数据缓冲一小段时间并发送到 APM server。

#!/bin/bash
cs=`echo ../lib/*jar | sed 's/ /:/g'`
export JAVA_OPTS='-javaagent:/app/agent/elastic-apm-agent-1.4.0.jar  -Delastic.apm.service_name=USER-CENTER -Delastic.apm.server_url=http://192.168.235.129:8200 -Delastic.apm.secret_token= -Delastic.apm.application_packages=com.open.capacity  -XX:+UseG1GC   -Xmx400M -Xms400M   -XX:MaxMetaspaceSize=128M   -XX:MetaspaceSize=128M      -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=45    -XX:+ParallelRefProcEnabled  -XX:MaxTenuringThreshold=3   -XX:+AlwaysPreTouch  -XX:+UseStringDeduplication -XX:StringDeduplicationAgeThreshold=3 -XX:-OmitStackTraceInFastThrow -Djava.security.egd=file:/dev/./urandom -verbose:gc  -XX:+PrintGCDetails   -XX:+PrintGCDateStamps  -XX:+PrintGCApplicationStoppedTime  -XX:+PrintGCApplicationConcurrentTime  -XX:+PrintTenuringDistribution -XX:+PrintClassHistogramBeforeFullGC  -XX:+PrintClassHistogramAfterFullGC -Xloggc:/tmp/logs/gc_%p.log  -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=/tmp/logs   -XX:ErrorFile=/tmp/logs/hs_error_pid%p.log -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=2199 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false '

nohup java -server  $JAVA_OPTS  -cp  .:$cs  com.open.capacity.UserCenterApp   &>/dev/null  &   echo $! >pid&

#java -server  $JAVA_OPTS  -cp  .:$cs  com.open.capacity.UserCenterApp     &   echo $! >pid&