ElasticSearch02索引一个文档视角看集群

本文是想透过一个空的elasticsearch实例应用了解创建一个索引在实体文件中占用多少磁盘空间, 所以在落实创建之前:

操作之前做一些记录;
利用python来创建一个索引;
观察索引创建后每个环节发生的变化;
激活另外一个空节点, 并观察数据同步情况;
总结。

操作之前做一些记录

文件分布

[root@es-master data]# tree /usr/local/elasticsearch-5.5.1/data/
/usr/local/elasticsearch-5.5.1/data/
└── nodes                       # 目录
    └── 0                       # 目录
        ├── node.lock           # 文件
        └── _state              # 目录
            ├── global-16.st    # 文件
            └── node-17.st      # 文件

3 directories, 3 files

文件大小(bytes)

[root@es-master data]# find /usr/local/elasticsearch-5.5.1/data/ -type f | xargs du -lsb
0   /usr/local/elasticsearch-5.5.1/data/nodes/0/node.lock
71  /usr/local/elasticsearch-5.5.1/data/nodes/0/_state/node-17.st
140 /usr/local/elasticsearch-5.5.1/data/nodes/0/_state/global-16.st

节点信息

[root@es-master data]# curl -X GET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "HawkEye",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

分片和索引的统计信息

[root@es-master data]# curl -X GET http://localhost:9200/_stats?pretty
{
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "_all" : {
    "primaries" : { },
    "total" : { }
  },
  "indices" : { }
}

利用python来创建一个索引

_01_create_index.py

# -.- coding:utf-8 -.-
from elasticsearch import Elasticsearch

es = Elasticsearch(hosts=["192.168.1.132"])
es.index(
    index="website",
    doc_type="blog",
    id=1,
    body={
        "title": "My first blog entry",
        "text":  "Just trying this out...",
        "date":  "2014/01/01"
    }
)

# 执行python文件, 落实索引创建
[root@es-master ~]# python _01_create_index.py

# 检查索引是否创建成功

[root@es-master data]# curl -XGET http://localhost:9200/website/blog/1?pretty
{
  "_index" : "website",
  "_type" : "blog",
  "_id" : "1",
  "_version" : 1,
  "found" : true,
  "_source" : {
    "date" : "2014/01/01",
    "text" : "Just trying this out...",
    "title" : "My first blog entry"
  }
}

观察索引创建后每个环节发生的变化

elasticsearch应用日志

[2017-08-18T22:41:03,598][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [website] creating index, cause [auto(bulk api)], templates [ mappings []
[2017-08-18T22:41:04,490][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [website/YhxMl-cOQUq6WCYH1sKjRg] create_mapping [blog]

文件分布

[root@es-master data]# tree /usr/local/elasticsearch-5.5.1/data/
/usr/local/elasticsearch-5.5.1/data/
└── nodes
    └── 0
        ├── indices
        │   └── YhxMl-cOQUq6WCYH1sKjRg
        │       ├── 0
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 1
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 2
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       ├── 3
        │       │   ├── index
        │       │   │   ├── _0.cfe
        │       │   │   ├── _0.cfs
        │       │   │   ├── _0.si
        │       │   │   ├── segments_3
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-2.tlog
        │       │       └── translog.ckp
        │       ├── 4
        │       │   ├── index
        │       │   │   ├── segments_1
        │       │   │   └── write.lock
        │       │   ├── _state
        │       │   │   └── state-0.st
        │       │   └── translog
        │       │       ├── translog-1.tlog
        │       │       └── translog.ckp
        │       └── _state
        │           └── state-4.st
        ├── node.lock
        └── _state
            ├── global-16.st
            └── node-17.st

26 directories, 32 files

文件大小(bytes)

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
632 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-4.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-master data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7272        # 存储总量(Bytes)  ==  7K大小

节点信息

[root@es-master data]# curl -X GET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "HawkEye",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 5,
  "active_shards" : 5,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

分片和索引的统计信息

[root@es-master data]# curl -X GET http://localhost:9200/_stats/indexing,store?pretty
{
  "_shards" : {
    "total" : 10,
    "successful" : 5,
    "failed" : 0
  },
  "_all" : {
    "primaries" : {
      "store" : {
        "size_in_bytes" : 5304,
        "throttle_time_in_millis" : 0
      },
      "indexing" : {
        "index_total" : 1,
        "index_time_in_millis" : 90,
        "index_current" : 0,
        "index_failed" : 0,
        "delete_total" : 0,
        "delete_time_in_millis" : 0,
        "delete_current" : 0,
        "noop_update_total" : 0,
        "is_throttled" : false,
        "throttle_time_in_millis" : 0
      }
    },
    "total" : {
      "store" : {
        "size_in_bytes" : 5304,
        "throttle_time_in_millis" : 0
      },
      "indexing" : {
        "index_total" : 1,
        "index_time_in_millis" : 90,
        "index_current" : 0,
        "index_failed" : 0,
        "delete_total" : 0,
        "delete_time_in_millis" : 0,
        "delete_current" : 0,
        "noop_update_total" : 0,
        "is_throttled" : false,
        "throttle_time_in_millis" : 0
      }
    }
  },
  "indices" : {
    "website" : {
      "primaries" : {
        "store" : {
          "size_in_bytes" : 5304,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 1,
          "index_time_in_millis" : 90,
          "index_current" : 0,
          "index_failed" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        }
      },
      "total" : {
        "store" : {
          "size_in_bytes" : 5304,
          "throttle_time_in_millis" : 0
        },
        "indexing" : {
          "index_total" : 1,
          "index_time_in_millis" : 90,
          "index_current" : 0,
          "index_failed" : 0,
          "delete_total" : 0,
          "delete_time_in_millis" : 0,
          "delete_current" : 0,
          "noop_update_total" : 0,
          "is_throttled" : false,
          "throttle_time_in_millis" : 0
        }
      }
    }
  }
}

激活另外一个空节点，并观察数据同步情况

主节点应用日志

[2017-08-18T23:38:13,202][INFO ][o.e.c.s.ClusterService   ] [node-1] added {{y1FxQfR}{y1FxQfRnTpyKdtid8ISeLQ}{lJd8BNRhQ5ywFkjLhZ2bPQ}{192.168.1.104}{192.168.1.104:9300},}, reason: zen-disco-node-join[{y1FxQfR}{y1FxQfRnTpyKdtid8ISeLQ}{lJd8BNRhQ5ywFkjLhZ2bPQ}{192.168.1.104}{192.168.1.104:9300}]
[2017-08-18T23:38:14,038][WARN ][o.e.d.z.ElectMasterService] [node-1] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2])
[2017-08-18T23:38:16,294][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[website][4]] ...]).

从节点应用日志

[2017-08-18T23:38:38,303][INFO ][o.e.d.DiscoveryModule    ] [y1FxQfR] using discovery type [zen]
[2017-08-18T23:38:40,023][INFO ][o.e.n.Node               ] initialized
[2017-08-18T23:38:40,023][INFO ][o.e.n.Node               ] [y1FxQfR] starting ...
[2017-08-18T23:38:40,420][INFO ][o.e.t.TransportService   ] [y1FxQfR] publish_address {192.168.1.104:9300}, bound_addresses {[::]:9300}
[2017-08-18T23:38:40,428][INFO ][o.e.b.BootstrapChecks    ] [y1FxQfR] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-08-18T23:38:43,810][INFO ][o.e.c.s.ClusterService   ] [y1FxQfR] detected_master {node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300}, added {{node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300},}, reason: zen-disco-receive(from master [master {node-1}{VlKvChmBR7aQhObWv_e4Kw}{cVyIHfX9RI6glUw3YsLEVg}{192.168.1.132}{192.168.1.132:9300} committed version [26]])
[2017-08-18T23:38:44,653][INFO ][o.e.h.n.Netty4HttpServerTransport] [y1FxQfR] publish_address {192.168.1.104:9200}, bound_addresses {[::]:9200}
[2017-08-18T23:38:44,653][INFO ][o.e.n.Node               ] [y1FxQfR] started

从节点数据目录结构

[root@es-slave01 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-6.st
140 ./nodes/0/_state/global-6.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-5.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_4
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-slave01 data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7387

大小发生了变化，所以要再看一下主节点的数据目录结构

主节点数据目录结构

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

[root@es-master data]# echo `find ./ -type f | xargs du -lsb | awk '{print $1}'` | tr " " "+" | bc
7387

集群生效后, 两个节点的数据完全一致, 这其实不是我想要看到的结果, 因为数据完全是镜像同步, 如果增加更多机器也是这种效果，那么就相当于是我所有的数据只存放在一台机器上一样，无非是多了灾备的情况和更高效的读，存储成本太高。
所以我需要多起两台空节点加入到集群中，然后再观察，发现数据存储方式发生了变化，elasticsearch内部将数据平摊到每个节点上了.

es-master

[root@es-master data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-17.st
140 ./nodes/0/_state/global-16.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-2.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-13.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_1
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

es-slave01

[root@es-slave01 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-6.st
140 ./nodes/0/_state/global-6.st
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/write.lock
405 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfe
3620    ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.cfs
375 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/_0.si
256 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/index/segments_4
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/3/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/_state/state-0.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/4/translog/translog-1.tlog

es-slave02

[root@es-slave02 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-10.st
140 ./nodes/0/_state/global-9.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/1/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-4.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_2
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog

es-slave03

[root@es-slave03 data]# find ./ -type f | xargs du -lsb
0   ./nodes/0/node.lock
71  ./nodes/0/_state/node-3.st
141 ./nodes/0/_state/global-3.st
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/2/translog/translog-1.tlog
0   ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/write.lock
162 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/index/segments_3
134 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/_state/state-1.st
48  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog.ckp
43  ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/0/translog/translog-1.tlog
747 ./nodes/0/indices/YhxMl-cOQUq6WCYH1sKjRg/_state/state-1.st

总结

四个节点, 仔细看下来shards分片们分别分布在:
es-mater: 1 3 4
es-slave01: 3 4
es-slave02: 0 1 2
es-slave03: 2 0

五个节点:
es-master: 0 1
es-slave01: 3 4
es-slave02: 2 4
es-slave03: 2 3
es-slave04: 0 1

六个节点:
es-master: 0 1
es-slave01: 3 4
es-slave02: 2 4
es-slave03: 2 3
es-slave04: 1
es-slave05: 0

综合下来看，数据要求至少是一份主分片和一份副本分片，并且散落在不同的机器上。

补充(提高性能)

正常情况下, 一个索引(index)会拥有五个分片(shards)，当机器数量达到5时，每个机器上都会均匀的分配到一个分片和一个副本分片。但是当拥有十台机器时，每个机器上就只会有一个分片(可能是主分片也可能是副本分片但是最终只有一份)，性能理论上就会得到大大的提升。
如果这还不够，那么可以在加5台机器，并在/webiste/_settings中设置number_of_replicas为2(默认是一份副本)，那么15台机器中将会有10台机器是副本分片5台机器是主分片，理论上来讲性能又会得到更多提升，同时灾备更健壮。
以此类推，20台机器可以将副本分片设置为3，25台机器可以将副本分片设置为4等等。

参考

-[x] 集群内的原理中文、英文

最后编辑于：2017.12.10 02:42:04

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 202,802评论 5赞 476
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 85,109评论 2赞 379
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 149,683评论 0赞 335
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 54,458评论 1赞 273
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 63,452评论 5赞 364
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 48,505评论 1赞 281
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 37,901评论 3赞 395
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 36,550评论 0赞 256
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 40,763评论 1赞 296
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 35,556评论 2赞 319
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 37,629评论 1赞 329
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,330评论 4赞 318
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 38,898评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 29,897评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,140评论 1赞 259
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 42,807评论 2赞 349
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,339评论 2赞 342

ElasticSearch02索引一个文档视角看集群

操作之前做一些记录

文件分布

文件大小(bytes)

节点信息

分片和索引的统计信息

利用python来创建一个索引

观察索引创建后每个环节发生的变化

elasticsearch应用日志

文件分布

文件大小(bytes)

节点信息

分片和索引的统计信息

激活另外一个空节点，并观察数据同步情况

主节点应用日志

从节点应用日志

从节点数据目录结构

主节点数据目录结构

es-master

es-slave01

es-slave02

es-slave03

总结

补充(提高性能)

参考

推荐阅读更多精彩内容