案例说明：
1、 通过sys_basebackup创建新备库。
2、 将备库加入到Cluster nodes管理，可以用kingbase_monitor.sh一键启停。
3、 主备复制切换测试。
此次测试案例用于kingbaseES V8R3读写分离集群在线扩容测试，主要分为三个步骤：
操作系统和数据库版本：
1）操作系统环境
[kingbase@#localhost ~]$ cat /etc/centos-release
CentOS Linux release 7.6.1810 (Core)2）数据库版本
TEST=# select version();
version
-------------------------------------------------------------------------------
Kingbase V008R003C002B0100 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-46), 64-bit
(1 row)
前期准备：
1、 原有节点，主库(192.168.4.127)、备库(192.168.4.129)，新增节点(192.168.4.159)。
2、 在新增节点安装相同版本的kingbaseES数据库。
3、 配置新增节点到原节点的ssh信任关系（root、kingbase）。
4、 关闭新增节点防火墙和selinux并启动crond服务。
一、在线增加数据库复制节点
1.1 查看kingbase cluster节点进程
首先确定原主备库的Cluster和kingbaseES服务状态正常。
主库：
kingbase@srv127 bin]$ ps -ef |grep kingbase
kingbase 15845 1 0 15:11 ? 00:00:00 /home/kingbase/cluster/ESHA/db/bin/kingbase -D /home/kingbase/cluster/ESHA/db/data
kingbase 15877 15845 0 15:11 ? 00:00:00 kingbase: logger process
kingbase 15879 15845 0 15:11 ? 00:00:00 kingbase: checkpointer process
kingbase 15880 15845 0 15:11 ? 00:00:00 kingbase: writer process
kingbase 15881 15845 0 15:11 ? 00:00:00 kingbase: wal writer process
kingbase 15882 15845 0 15:11 ? 00:00:00 kingbase: autovacuum launcher process
kingbase 15883 15845 0 15:11 ? 00:00:00 kingbase: archiver process
kingbase 15884 15845 0 15:11 ? 00:00:00 kingbase: stats collector process
kingbase 15885 15845 0 15:11 ? 00:00:00 kingbase: bgworker: syslogical supervisor
kingbase 16851 15845 0 15:11 ? 00:00:00 kingbase: wal sender process SYSTEM 192.168.4.127(52967) streaming 0/180000D0
root 17781 1 0 15:11 ? 00:00:00 ./kingbasecluster -n
root 17830 17781 0 15:11 ? 00:00:00 kingbasecluster: watchdog
root 17895 17781 0 15:11 ? 00:00:00 kingbasecluster: lifecheck
root 17897 17895 0 15:11 ? 00:00:00 kingbasecluster: heartbeat receiver
root 17898 17895 0 15:11 ? 00:00:00 kingbasecluster: heartbeat sender
root 17899 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17900 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17901 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17912 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17913 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17914 17781 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 17924 17781 0 15:11 ? 00:00:00 kingbasecluster: PCP: wait for connection request
root 17925 17781 0 15:11 ? 00:00:00 kingbasecluster: worker process
kingbase 20397 15845 0 15:12 ? 00:00:00 kingbase: SUPERMANAGER_V8ADMIN TEST 192.168.4.127(53139) idle
备库：
[kingbase@srv129 bin]$ ps -ef |grep kingbase
kingbase 30708 1 0 15:11 ? 00:00:00 /home/kingbase/cluster/ESHA/db/bin/kingbase -D /home/kingbase/cluster/ESHA/db/data
kingbase 30709 30708 0 15:11 ? 00:00:00 kingbase: logger process
kingbase 30710 30708 0 15:11 ? 00:00:00 kingbase: startup process recovering 000000030000000000000018
kingbase 30714 30708 0 15:11 ? 00:00:00 kingbase: checkpointer process
kingbase 30715 30708 0 15:11 ? 00:00:00 kingbase: writer process
kingbase 30716 30708 0 15:11 ? 00:00:00 kingbase: stats collector process
kingbase 30995 1 0 10月18 ? 00:00:17 /home/kingbase/cluster/KHA6/KHA/kingbase/bin/kbha -A daemon -f /home/kingbase/cluster/KHA6/KHA/kingbase/bin/../etc/repmgr.conf
kingbase 31173 30708 0 15:11 ? 00:00:00 kingbase: wal receiver process streaming 0/180000D0
root 31222 1 0 15:11 ? 00:00:00 ./kingbasecluster -n
root 31264 31222 0 15:11 ? 00:00:00 kingbasecluster: watchdog
root 31396 31222 0 15:11 ? 00:00:00 kingbasecluster: lifecheck
root 31398 31396 0 15:11 ? 00:00:00 kingbasecluster: heartbeat receiver
root 31399 31396 0 15:11 ? 00:00:00 kingbasecluster: heartbeat sender
root 31400 31222 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
root 31402 31222 0 15:11 ? 00:00:00 kingbasecluster: wait for connection request
……
root 31421 31222 0 15:11 ? 00:00:00 kingbasecluster: PCP: wait for connection request
root 31422 31222 0 15:11 ? 00:00:00 kingbasecluster: worker process
kingbase 32062 30708 0 15:12 ? 00:00:00 kingbase: SUPERMANAGER_V8ADMIN TEST 192.168.4.127(19332) idle
1.2 查看主备复制信息
保证集群中主备复制状态正常。
kingbase@srv127 ~]$ ksql -U SYSTEM -W 123456 TEST
警告: License file will expire in 1 days.
ksql (V008R003C002B0160)
Type "help" for help.TEST=# select * from sys_stat_replication;
pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_lo
cation | write_location | flush_location | replay_location | sync_priority | sync_state
-------+----------+---------+------------------+---------------+-------------
16851 | 10 | SYSTEM | srv129 | 192.168.4.129 | | 52967 | 2020-10-23 15:11:17.836557+08 | | streaming | 0/18000
0D0 | 0/180000D0 | 0/180000D0 | 0/180000D0 | 0 | async
(1 row)
1.3 在线增加新的备库节点
为保证测试环境和接近于生产环境，在创建备库时，在主库端进行大的事务处理，当备库创建完成后查看是否合主库数据一致。
在主库进行事务处理（模拟生产库）：
prod=# insert into t10 values (generate_series(1,10000000),'tom');prod=# select count(*) from t10;
count
----------
11000000
(1 row)
1.3.1 在新备库上通过sys_basebackup在线创建备库
1） 创建备库数据存储目录
注意：目录位置尽量和其他已存在节点的目录位置一致，这样对于后期配置改动较少。
[kingbase@srv159 v8r3]$ mkdir -p /home/kingbase/cluster/ESHA/db/data[kingbase@srv159 v8r3]$ chmod 0700 /home/kingbase/cluster/ESHA/db/data
2）通过sys_basebackup创建备库
[kingbase@srv159 v8r3]$sys_basebackup -h 192.168.4.127 -D /home/kingbase/cluster/ESHA/db/data -F p -x -v -P -U SYSTEM -W 123456 -p 54321事务日志起始于时间点: 0/B20060D0, 基于时间表32079583/2079583 kB (100%), 1/1 表空间transaction log end point: 0/D5C48538sys_basebackup: base backup completed
** 1.3.2 配置备库recovery.conf文件**
[kingbase@srv159 data]$ pwd/home/kingbase/cluster/ESHA/db/data[kingbase@srv159 data]$ cat recovery.confstandby_mode='on'primary_conninfo='port=54321 host=192.168.4.127 user=SYSTEM password=MTIzNDU2 application_name=srv159'recovery_target_timeline='latest'primary_slot_name ='slot_srv159'
1.3.3 配置备库kingbase.conf文件（部分内容)
# Add settings for extensions heremax_wal_senders=4wal_keep_segments=256hot_standby_feedback=onmax_prepared_transactions=100port=54321control_file_copy='/home/kingbase/cluster/ESHA/template.bk'fsync=onwal_log_hints=onarchive_mode=onhot_standby=onwal_level=replica###synchronous_standby_names='1(srv129)'archive_dest='/home/kingbase/cluster/ESHA/archivedir/'max_replication_slots=4log_directory='/home/kingbase/cluster/ESHA/db/data/sys_log/'
1.3.4 配置所有节点sys_hba.conf文件
# "local" is for Unix domain socket connections onlylocal all all md5# IPv4 local connections:host all all 127.0.0.1/32 md5host all all 0.0.0.0/0 md5# IPv6 local connections:host all all ::1/128 md5host all all ::0/0 md5# Allow replication connections from localhost, by a user with the# replication privatehost all SYSTEM 192.168.4.127/16 md5host all SYSTEM 192.168.4.129/16 md5host all SYSTEM 192.168.4.159/16 md5 host replication SYSTEM 192.168.4.127/16 md5host replication SYSTEM 192.168.4.129/16 md5host replication SYSTEM 192.168.4.159/16 md5
注意：修改已有节点的sys_hba.conf文件后，需要reload数据库进程才能使配置生效。
[kingbase@srv129 data]$ sys_ctl reload -D /home/kingbase/cluster/ESHA/db/data服务器进程发出信号
1.3.5 从主库拷贝以下目录和文件到新备库
通过scp远程拷贝以下目录和文件到新增节点相同的位置：
说明：“/home/kingbase/cluster/ESHA”是集群部署的目录，其中“ESHA”是集群名称。对于archivedir也可以创建空目录。
[kingbase@#localhost ESHA]$ pwd/home/kingbase/cluster/ESHA[kingbase@#localhost ESHA]$ ls -lh总用量 8.0Kdrwxrwxr-x 2 kingbase kingbase 6 10月 30 14:23 archivedirdrwxrwxr-x 7 kingbase kingbase 84 10月 30 17:00 dbdrwxrwxr-x 3 kingbase kingbase 181 10月 30 17:23 logdrwxrwxr-x 3 kingbase kingbase 29 10月 30 14:55 run-rw------- 1 kingbase kingbase 8.0K 10月 31 13:19 template.bk [kingbase@srv127 db]$ pwd/home/kingbase/cluster/ESHA/db[kingbase@srv127 db]$ ls -lh总用量 20Kdrwxrwxr-x 2 kingbase kingbase 4.0K 10月 29 17:29 bindrwxrwxr-x 2 kingbase kingbase 295 10月 29 17:26 etc-rw------- 1 kingbase kingbase 151 8月 18 20:13 kingbase.logdrwxrwxr-x 2 kingbase kingbase 4.0K 7月 27 10:41 libdrwxrwxr-x 9 kingbase kingbase 4.0K 7月 27 10:41 share
注意：目录和文件拷贝不要拷贝kingbasecluster目录到新增节点，在V8R3的集群中，cluster只支持主备两个节点，新增节点只能是data node。
拷贝主库/etc/cron.d/KINGBASECRON文件到备库相同目录下：
1.3.6 在主库端添加replication_slot
1） 查看已经创建的replication_slot:
TEST=# select * from sys_replication_slots; slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn-------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+--------------------- slot_srv127 | | physical | | | f | | | | | slot_srv129 | | physical | | | f | | | | |(2 rows)
2） 创建新增备库的replication_slot
TEST=# select * from sys_create_physical_replication_slot('slot_srv159'); slot_name | xlog_position-------------+--------------- slot_srv159 |(1 row)
3）查看主备复制的replication_slots
TEST=# select * from sys_replication_slots; slot_name | plugin | slot_type | datoid | database | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn-------------+--------+-----------+--------+----------+--------+------------+------+--------------+-------------+--------------------- slot_srv127 | | physical | | | f | | | | | slot_srv129 | | physical | | | f | | | | | slot_srv159 | | physical | | | f | | | | |(3 rows)
** 1.3.7 启动备库实例加入主备复制架构**
[kingbase@srv159 data]$ sys_ctl start -D /home/kingbase/cluster/ESHA/db/data正在启动服务器进程[kingbase@srv159 data]$ 日志: 日志输出重定向到日志收集进程提示: 后续的日志输出将出现在目录 "/home/kingbase/cluster/ESHA/db/data/sys_log"中. 查看备库进程：[kingbase@srv159 data]$ ps -ef |grep kingbasekingbase 5071 1 0 20:01 pts/3 00:00:00 /opt/Kingbase/ES/V8R3/Server/bin/kingbase -D /home/kingbase/cluster/ESHA/db/datakingbase 5072 5071 0 20:01 ? 00:00:00 kingbase: logger processkingbase 5073 5071 90 20:01 ? 00:00:19 kingbase: startup process waiting for 0000000300000000000000D6kingbase 5077 5071 0 20:01 ? 00:00:00 kingbase: checkpointer processkingbase 5078 5071 0 20:01 ? 00:00:00 kingbase: writer processkingbase 5199 5071 0 20:01 ? 00:00:00 kingbase: stats collector processkingbase 5200 5071 0 20:01 ? 00:00:00 kingbase: wal receiver processkingbase 5201 5071 0 20:01 ? 00:00:00 kingbase: wal sender process SYSTEM 192.168.4.159(62391) idle
** 1.3.8 查看主备复制信息**
1）查看主备复制状态信息
TEST=# select * from sys_stat_replication; pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state-------+----------+---------+------------------+---------------+------------- 16851 | 10 | SYSTEM | srv129 | 192.168.4.129 | | 52967 | 2020-10-23 15:11:17.836557+08 | | streaming | 0/ED000000 | 0/ED000000 | 0/ED000000 | 0/ECFFF418 | 0 | async 12202 | 10 | SYSTEM | srv159 | 192.168.4.159 | | 47993 | 2020-10-23 20:10:32.521960+08 | 2059 | streaming | 0/ED000000 | 0/ED000000 | 0/ED000000 | 0/ECFFF418 | 0 | async(2 rows)
2）查看数据库数据信息
主库：prod=# select count(*) from t10; count---------- 21000000(1 row) 备库：prod=# select count(*) from t10; count---------- 21000000(1 row)
备注：
1） 在主备状态信息中已经查看到新增节点（srv159）的复制状态信息。2） 查看主库和备库同步数据一致，说明在创建基础备份期间，备库和主库保持了数据的一致性。3） 由以上可知，备库创建完成。
二、配置新节点加入集群服务一键启停
2.1 编辑所有节点HAmodule.conf
注意：在KB_ALL_IP参数中加入新的备库节点的ip。
2.2 在新增节点修改HAmodule.conf
注意：将KB_LOCALHOST_IP参数配置为本机ip。
2.3 配置已经存在节点（Cluster的主备库）kingbase_cluster.conf文件
注意：图中黄色部分为新增节点的配置，仔细修改，否则在show pool_nodes时无法发现新增节点。
2.4 测试kingbase_monitor.sh一键启停集群：
[kingbase@srv129 bin]$ ./kingbase_monitor.sh restart-----------------------------------------------------------------------2020-10-29 17:32:33 KingbaseES automation beging...2020-10-29 17:32:33 stop kingbasecluster [192.168.4.127] ...DEL VIP NOW AT 2020-10-29 17:32:32 ON ens192No VIP on my dev, nothing to do.2020-10-29 17:32:33 Done...2020-10-29 17:32:33 stop kingbasecluster [192.168.4.129] ...DEL VIP NOW AT 2020-10-29 17:32:34 ON ens192No VIP on my dev, nothing to do.2020-10-29 17:32:34 Done...2020-10-29 17:32:34 stop kingbase [192.168.4.127] ...set /home/kingbase/cluster/ESHA/db/data down now...2020-10-29 17:32:37 Done...2020-10-29 17:32:38 Del kingbase VIP [192.168.4.99/24] ...DEL VIP NOW AT 2020-10-29 17:32:37 ON ens192No VIP on my dev, nothing to do.2020-10-29 17:32:38 Done...2020-10-29 17:32:38 stop kingbase [192.168.4.129] ...set /home/kingbase/cluster/ESHA/db/data down now...2020-10-29 17:32:40 Done...2020-10-29 17:32:41 Del kingbase VIP [192.168.4.99/24] ...DEL VIP NOW AT 2020-10-29 17:32:42 ON ens192execute: [/sbin/ip addr del 192.168.4.99/24 dev ens192]Oprate del ip cmd end.2020-10-29 17:32:42 Done...2020-10-29 17:32:42 stop kingbase [192.168.4.159] ...2020-10-29 17:32:42 Done...2020-10-29 17:32:43 Del kingbase VIP [192.168.4.99/24] ...DEL VIP NOW AT 2020-10-29 17:32:43 ON ens192No VIP on my dev, nothing to do.2020-10-29 17:32:43 Done.........................all stop.. start crontab kingbase position : [3]Redirecting to /bin/systemctl restart crond.servicestart crontab kingbase position : [3]Redirecting to /bin/systemctl restart crond.serviceADD VIP NOW AT 2020-10-29 17:32:51 ON ens192execute: [/sbin/ip addr add 192.168.4.99/24 dev ens192 label ens192:2]execute: /usr/sbin/arping -U 192.168.4.99 -I ens192 -w 1ARPING 192.168.4.99 from 192.168.4.99 ens192Sent 1 probes (1 broadcast(s))Received 0 response(s)Redirecting to /bin/systemctl restart crond.servicewait kingbase recovery 5 sec...start crontab kingbasecluster line number: [4]Redirecting to /bin/systemctl restart crond.servicestart crontab kingbasecluster line number: [4]Redirecting to /bin/systemctl restart crond.service......................all started.....now we check again==================================================================| ip | program| [status][ 192.168.4.127]| [kingbasecluster]| [active][ 192.168.4.129]| [kingbasecluster]| [active][ 192.168.4.127]| [kingbase]| [active][ 192.168.4.129]| [kingbase]| [active][ 192.168.4.159]| [kingbase]| [active]==================================================================
注意：从以上可以获得，集群架构为2个节点的cluster，3个节点的data node。
2.5 查看新增节点信息
三、集群切换测试
3.1 测试目标
当原备库数据库服务停止或宕机，在主库服务或宕机时，新的备库切换为主库。
3.2 测试步骤
先停止原备库的kingbaseES服务。再停止主库的kingbaseES服务。查看新的备库是否被切换为主库。
3.3 实施主备切换
[kingbase@srv129 bin]$ sys_ctl stop -D /home/kingbase/cluster/ESHA/db/data等待服务器进程关闭 .... 完成服务器进程已经关闭
1）停止原备库的kingbaseES服务
[kingbase@srv129 bin]$ sys_ctl stop -D /home/kingbase/cluster/ESHA/db/data等待服务器进程关闭 .... 完成服务器进程已经关闭
2）查看复制状态信息
1）复制状态信息
TEST=# select * from sys_stat_replication; pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state------+----------+---------+------------------+---------------+-------------- 1556 | 10 | SYSTEM | srv129 | 192.168.4.129 | | 20766 | 2020-10-31 15:17:03.946716+08 | | streaming | 1/5603A788 | 1/5603A788 | 1/5603A788 | 1/5603A750 | 1 | sync 3387 | 10 | SYSTEM | srv159 | 192.168.4.159 | | 47333 | 2020-10-31 15:21:05.275911+08 | | streaming | 1/5603A788 | 1/5603A788 | 1/5603A788 | 1/5603A750 | 2 | potential(2 rows)
注意：在一主两从的架构中，新增节点的sync_stat状态为”potential”
2）原备库kingbaseES被停止
TEST=# select * from sys_stat_replication; pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state------+----------+---------+------------------+---------------+--------------- 3387 | 10 | SYSTEM | srv159 | 192.168.4.159 | | 47333 | 2020-10-31 15:21:05.275911+08 | | streaming | 1/5603A980 | 1/5603A980 | 1/5603A980 | 1/5603A8D8 | 0 | async(1 row)
注意：在一主一从的架构中，新增节点的sync_stat状态为“async”。
3）查看pool node信息
TEST=# show pool_nodes; node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay---------+---------------+-------+--------+-----------+---------+------------+-------------------+------------------- 0 | 192.168.4.127 | 54321 | up | 0.333333 | primary | 5 | false | 0 1 | 192.168.4.129 | 54321 | down | 0.333333 | standby | 0 | false | 0 2 | 192.168.4.159 | 54321 | up | 0.333333 | standby | 0 | true | 0(3 rows)
3）停止主库kingbaseES数据库服务
[kingbase@srv127 etc]$ sys_ctl stop -D /home/kingbase/cluster/ESHA/db/data等待服务器进程关闭 .... 完成服务器进程已经关闭
** 4）切换完成查看备库信息**
1）查看新备库的kingbaseES进程
[kingbase@#localhost log]$ ps -ef |grep kingbasekingbase 11680 1 0 15:21 pts/1 00:00:01 /home/kingbase/cluster/ESHA/db/bin/kingbase -D /home/kingbase/cluster/ESHA/db/datakingbase 11681 11680 0 15:21 ? 00:00:00 kingbase: logger processkingbase 11686 11680 0 15:21 ? 00:00:00 kingbase: checkpointer processkingbase 11687 11680 0 15:21 ? 00:00:00 kingbase: writer processkingbase 11689 11680 0 15:21 ? 00:00:00 kingbase: stats collector processkingbase 14821 31212 0 10月30 pts/0 00:00:00 ../bin/kingbasecluster -n -dkingbase 14840 14821 0 10月30 pts/0 00:00:35 kingbasecluster: watchdogroot 14850 14675 0 09:48 pts/1 00:00:00 su - kingbasekingbase 14851 14850 0 09:48 pts/1 00:00:00 -bashkingbase 15518 11680 0 15:33 ? 00:00:00 kingbase: wal writer processkingbase 15519 11680 0 15:33 ? 00:00:00 kingbase: autovacuum launcher processkingbase 15520 11680 0 15:33 ? 00:00:00 kingbase: archiver process last was 00000005.historykingbase 15521 11680 0 15:33 ? 00:00:00 kingbase: bgworker: syslogical supervisorkingbase 16075 11680 0 15:35 ? 00:00:00 kingbase: SUPERMANAGER_V8ADMIN TEST 192.168.4.127(34260) idle
2）连接实例查看数据库状态
[kingbase@#localhost data]$ ksql -U system -W 123456 TESTksql (V008R003C002B0160)Type "help" for help. TEST=# select sys_is_in_recovery(); sys_is_in_recovery-------------------- f(1 row)
注意：备库已经切换为主库状态。
3）查看recovery日志
[kingbase@#localhost log]$ tail -f recovery.log
---------------------------------------------------------------------
2020-10-31 15:36:02 recover beging...
2020-10-31 15:36:02 check if the network is ok
ping trust ip 192.168.4.1 scuccess ping times :[3], scuccess times:[3]
determine if i am master or standby
I,m node is primary, determine whether there is a standby db can be set to synchronous standby
由以上可以获得，备库已经切换为主库。
四、总结
kingbaseES V8R3读写分离集群可以在线手工方式增加新的节点，进行集群架构的横向扩展。
在修改kingbasecluster.conf后需要重新启动kingbasecluster服务，可以先在cluster备节点通过命令(kingbasecluster –m fast stop关闭服务），然后再关闭cluster主节点服务；先启动cluster主节点服务(kingbasecluster –n –d>kingbasecluster.log 2>&1 &)方式启动集群服务，主节点启动完成后再启动cluster备节点。
增加节点前做好测试，应该在业务低峰时候来完成。
本测试文档只是在CentOS 7环境下，其他版本使用前请在线下测试。
