MySQL Group Replication的RECOVERING状态深度理解

接收到报警通知，db3这台组复制成员故障down机了，等修复好，启动服务器，然后再启动mysql实例，进去查看状态，变成了RECOVERING，如下所示；

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST          | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
| group_replication_applier | 3381d155-d7d1-11e6-94f7-b8ca3af6e36c | hch_test_dbm2_121_71 |        3317 | ONLINE       |
| group_replication_applier | 664b9ce9-d7de-11e6-9e8c-18a99b763071 | hch_test_web_1_24    |        3317 | ONLINE       |
| group_replication_applier | 84dba8ff-d7d2-11e6-aa9a-18a99b76310d | bpe_service          |        3317 | RECOVERING   |
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
3 rows in set (0.00 sec)


mysql>

这种状态有点类似mongodb的sendary的RECOVERING状态，对于刚刚出现的MySQL Group Replication新技术，遇到这种情况，这种情况怎么办？

在mongodb的这种模式下，一般是secondary从primary库上不停复制数据，所以我们遵循这种思路，去看下db2的数据目录，看到有很多relay-bin的recovery日志在应用执行，如下所示relay日志包记录

-rw-r----- 1 mysql mysql  383 1月  17 16:08 bpe_service-relay-bin-group_replication_applier.000019
-rw-r----- 1 mysql mysql  502 1月  17 16:11 bpe_service-relay-bin-group_replication_applier.000020
-rw-r----- 1 mysql mysql  502 1月  17 16:23 bpe_service-relay-bin-group_replication_applier.000021
-rw-r----- 1 mysql mysql  421 1月  17 16:23 bpe_service-relay-bin-group_replication_applier.000022
-rw-r----- 1 mysql mysql  228 1月  17 16:23 bpe_service-relay-bin-group_replication_applier.index
-rw-r----- 1 mysql mysql  312 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000007
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000008
-rw-r----- 1 mysql mysql  391 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000009
-rw-r----- 1 mysql mysql  312 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000010
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000011
-rw-r----- 1 mysql mysql  391 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000012
-rw-r----- 1 mysql mysql  312 1月  17 16:24 bpe_service-relay-bin-group_replication_recovery.000013
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000014
-rw-r----- 1 mysql mysql  391 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000015
-rw-r----- 1 mysql mysql  312 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000016
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000017
-rw-r----- 1 mysql mysql  391 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000018
-rw-r----- 1 mysql mysql  312 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000019
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000020
-rw-r----- 1 mysql mysql  391 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000021
-rw-r----- 1 mysql mysql  312 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000022
-rw-r----- 1 mysql mysql 1.1G 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000023
-rw-r----- 1 mysql mysql  391 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000024
-rw-r----- 1 mysql mysql  312 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.000025
-rw-r----- 1 mysql mysql 181M 1月  17 16:26 bpe_service-relay-bin-group_replication_recovery.000026
-rw-r----- 1 mysql mysql 1.2K 1月  17 16:25 bpe_service-relay-bin-group_replication_recovery.index
drwxr-x--- 2 mysql mysql 4.0K 1月  17 14:11 business_db

然后在mysql窗口，可以看到几个线程，其中有一个就是Reading event from the relay log线程，就是表示正在应用relay日志：

mysql> show full processlist;
+----+-------------+-----------+------+---------+------+--------------------------------------------------------+-----------------------+
| Id | User        | Host      | db   | Command | Time | State                                                  | Info                  |
+----+-------------+-----------+------+---------+------+--------------------------------------------------------+-----------------------+
| 28 | system user |           | NULL | Connect |  701 | Suspending                                             | NULL                  |
| 31 | system user |           | NULL | Connect |  701 | Slave has read all relay log; waiting for more updates | NULL                  |
| 35 | system user |           | NULL | Connect |  699 | Waiting for master to send event                       | NULL                  |
| 36 | system user |           | NULL | Connect | 4519 | Reading event from the relay log                       | NULL                  |
| 38 | root        | localhost | NULL | Query   |    0 | starting                                               | show full processlist |
+----+-------------+-----------+------+---------+------+--------------------------------------------------------+-----------------------+
5 rows in set (0.00 sec)


mysql>

再过一会儿，数据同步完成之后，bpe_service就会从RECOVERING变成ONLINE状态。这个更加证实了，数据已经同步完成。

mysql> SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
| CHANNEL_NAME              | MEMBER_ID                            | MEMBER_HOST          | MEMBER_PORT | MEMBER_STATE |
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
| group_replication_applier | 3381d155-d7d1-11e6-94f7-b8ca3af6e36c | hch_test_dbm2_121_71 |        3317 | ONLINE       |
| group_replication_applier | 664b9ce9-d7de-11e6-9e8c-18a99b763071 | hch_test_web_1_24    |        3317 | ONLINE       |
| group_replication_applier | 84dba8ff-d7d2-11e6-aa9a-18a99b76310d | bpe_service          |        3317 | ONLINE       |
+---------------------------+--------------------------------------+----------------------+-------------+--------------+
3 rows in set (0.00 sec)


mysql>

去查看后台error log日志显示，刚开始启动，在做CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'的时候，没有取到值，然后看后面同步完后，就会执行加入组复制成员的命令

“CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='hch_test_web_1_24', master_port= 3317, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''”，

在执行结束后，最后告诉我们“'This server was declared online within the replication group'”已经加入到了组成员，而且变成了实时的ONLINE状态：

2017-01-17T08:23:15.710146Z 4 [Note] Plugin group_replication reported: 'auto_increment_increment is reset to 1'
2017-01-17T08:23:15.710200Z 4 [Note] Plugin group_replication reported: 'auto_increment_offset is reset to 1'
2017-01-17T08:23:15.710401Z 20 [Note] Error reading relay log event for channel 'group_replication_applier': slave SQL thread was killed
2017-01-17T08:23:15.711212Z 17 [Note] Plugin group_replication reported: 'The group replication applier thread was killed'
2017-01-17T08:23:42.141239Z 4 [Note] Plugin group_replication reported: 'Group communication SSL configuration: group_replication_ssl_mode: "DISABLED"'
2017-01-17T08:23:42.141451Z 4 [Note] Plugin group_replication reported: '[GCS] Added automatically IP ranges 127.0.0.1/8,192.168.121.111/23 to the whitelist'
2017-01-17T08:23:42.141990Z 4 [Note] Plugin group_replication reported: '[GCS] Translated 'db2' to 192.168.121.111'
2017-01-17T08:23:42.142174Z 4 [Note] Plugin group_replication reported: '[GCS] SSL was not enabled'
2017-01-17T08:23:42.142220Z 4 [Note] Plugin group_replication reported: 'Initialized group communication with configuration: group_replication_group_name: "e4668cea-d7ca-11e6-86b5-18a99b76310d"; group_replication_local_address: "db2:24902"; group_replication_group_seeds: "db1:24901,db2:24902,db3:24903"; group_replication_bootstrap_group: false; group_replication_poll_spin_loops: 0; group_replication_compression_threshold: 1000000; group_replication_ip_whitelist: "AUTOMATIC"'
2017-01-17T08:23:42.142936Z 28 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_applier' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

-- #（1）尝试启动默认的组复制，结果信息不对称，启动失败，然后开始初始化Group Replication
2017-01-17T08:23:42.166956Z 31 [Note] Slave SQL thread for channel 'group_replication_applier' initialized, starting replication in log 'FIRST' at position 0, relay log './bpe_service-relay-bin-group_replication_applier.000019' position: 4
2017-01-17T08:23:42.166960Z 4 [Note] Plugin group_replication reported: 'Group Replication applier module successfully initialized!'
2017-01-17T08:23:42.167064Z 4 [Note] Plugin group_replication reported: 'auto_increment_increment is set to 7'
2017-01-17T08:23:42.167079Z 4 [Note] Plugin group_replication reported: 'auto_increment_offset is set to 12002'
2017-01-17T08:23:42.167219Z 0 [Note] Plugin group_replication reported: 'state 4257 action xa_init'
2017-01-17T08:23:42.167292Z 0 [Note] Plugin group_replication reported: 'Successfully bound to 0.0.0.0:24902 (socket=49).'
2017-01-17T08:23:42.167341Z 0 [Note] Plugin group_replication reported: 'Successfully set listen backlog to 32 (socket=49)!'
2017-01-17T08:23:42.167357Z 0 [Note] Plugin group_replication reported: 'Successfully unblocked socket (socket=49)!'
2017-01-17T08:23:42.167400Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.167585Z 0 [Note] Plugin group_replication reported: 'Ready to accept incoming connections on 0.0.0.0:24902 (socket=49)!'
2017-01-17T08:23:42.174427Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 50'
2017-01-17T08:23:42.174807Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.175008Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 63'
2017-01-17T08:23:42.175234Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.175411Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 65'
2017-01-17T08:23:42.175614Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.175799Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 67'
2017-01-17T08:23:42.176006Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.176176Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 69'
2017-01-17T08:23:42.176377Z 0 [Note] Plugin group_replication reported: 'connecting to db2 24902'
2017-01-17T08:23:42.176563Z 0 [Note] Plugin group_replication reported: 'client connected to db2 24902 fd 71'
2017-01-17T08:23:42.176876Z 0 [Note] Plugin group_replication reported: 'connecting to db1 24901'
2017-01-17T08:23:42.177312Z 0 [Note] Plugin group_replication reported: 'client connected to db1 24901 fd 73'
2017-01-17T08:23:43.463474Z 0 [Note] Plugin group_replication reported: 'state 4257 action xa_snapshot'
2017-01-17T08:23:43.463969Z 0 [Note] Plugin group_replication reported: 'new state x_recover'
2017-01-17T08:23:43.464000Z 0 [Note] Plugin group_replication reported: 'state 4277 action xa_complete'
2017-01-17T08:23:43.464275Z 0 [Note] Plugin group_replication reported: 'new state x_run'
2017-01-17T08:23:44.619112Z 0 [Note] Plugin group_replication reported: 'Starting group replication recovery with view_id 14844510167669342:17' 

--#（2） 开始进行组内数据recovery
2017-01-17T08:23:44.619451Z 34 [Note] Plugin group_replication reported: 'Establishing group recovery connection with a possible donor. Attempt 1/10'
2017-01-17T08:23:44.624181Z 34 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='hch_test_web_1_24', master_port= 3317, master_log_file='', master_log_pos= 4, master_bind=''.
2017-01-17T08:23:44.628853Z 34 [Note] Plugin group_replication reported: 'Establishing connection to a group replication recovery donor 664b9ce9-d7de-11e6-9e8c-18a99b763071 at hch_test_web_1_24 port: 3317.'
2017-01-17T08:23:44.629156Z 35 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2017-01-17T08:23:44.635870Z 36 [Note] Slave SQL thread for channel 'group_replication_recovery' initialized, starting replication in log 'FIRST' at position 0, relay log './bpe_service-relay-bin-group_replication_recovery.000001' position: 4
2017-01-17T08:23:44.681897Z 35 [Note] Slave I/O thread for channel 'group_replication_recovery': connected to master 'repl@hch_test_web_1_24:3317',replication started in log 'FIRST' at position 4
2017-01-17T08:43:34.219952Z 34 [Note] Plugin group_replication reported: 'Terminating existing group replication donor connection and purging the corresponding logs.'
2017-01-17T08:43:34.220609Z 36 [Note] Slave SQL thread for channel 'group_replication_recovery' exiting, replication stopped in log 'binlog.000042' at position 185511578
2017-01-17T08:43:34.226869Z 35 [Note] Slave I/O thread killed while reading event for channel 'group_replication_recovery'
2017-01-17T08:43:34.226886Z 35 [Note] Slave I/O thread exiting for channel 'group_replication_recovery', read up to log 'binlog.000042', position 185511578
2017-01-17T08:43:34.269973Z 34 [Note] 'CHANGE MASTER TO FOR CHANNEL 'group_replication_recovery' executed'. Previous state master_host='hch_test_web_1_24', master_port= 3317, master_log_file='', master_log_pos= 4, master_bind=''. New state master_host='<NULL>', master_port= 0, master_log_file='', master_log_pos= 4, master_bind=''.

-- # （3）已经数据恢复完成，加入组，成为组成员
2017-01-17T08:43:34.290920Z 0 [Note] Plugin group_replication reported: 'This server was declared online within the replication group'

小结：mysql group replication的使用来说，有些理念比如mongodb的分片以及primary-secondary的思路理念可以参考下，这样理解起来就会比较容易一些。

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 202,980评论 5赞 476
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 85,178评论 2赞 380
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 149,868评论 0赞 336
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 54,498评论 1赞 273
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 63,492评论 5赞 364
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 48,521评论 1赞 281
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 37,910评论 3赞 395
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 36,569评论 0赞 256
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 40,793评论 1赞 296
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 35,559评论 2赞 319
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 37,639评论 1赞 329
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 33,342评论 4赞 318
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 38,931评论 3赞 307
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 29,904评论 0赞 19
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,144评论 1赞 259
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 42,833评论 2赞 349
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 42,350评论 2赞 342

推荐阅读更多精彩内容