代码拉取完成，页面将自动刷新
【标题描述】:2.0.4/2.0.5(不带cm)灰度地升级5.0.1(带cm)，提交/回滚失败
【测试类型：工具功能】【测试版本：5.0.1】【升级】 2.0.4/2.0.5(不带cm)灰度地升级5.0.1(带cm)，提交/回滚失败
【操作系统和硬件信息】(查询命令: cat /etc/system-release, uname -a):
[upgrade_1130@kwepwebenv07953 bin]$ cat /etc/system-release
CentOS Linux release 7.6.1810 (Core)
[upgrade_1130@kwepwebenv07953 bin]$ uname -a
Linux kwepwebenv07953 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
【测试环境】(单机/1主x备x级联备):
一主两备
【被测功能】:
升级
【测试类型】:
功能
【数据库版本】(查询命令: gaussdb -V):
gsql (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
gsql (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
gsql (openGauss 5.0.1 build d855a18f) compiled at 2023-11-28 19:13:55 commit 0 last mr
【预置条件】:
不涉及
【操作步骤】(请填写详细的操作步骤):
1.CI配置直接升级路径及版本并执行
2.0.4-5.0.1 灰度升级 ncm-cm
2.0.4-5.0.1 灰度升级，回滚，再提交 ncm-cm
2.0.5-5.0.1 灰度升级，回滚，再提交 ncm-cm
【预期输出】:
升级成功，提交成功，回滚成功
【实际输出】:
问题复现路径：
2.0.5不带cm灰度升级到5.0.1带cm 回滚失败
2.0.4不带cm灰度升级到5.0.1带cm，提交失败
2.0.4不带cm灰度升级到5.0.1带cm，回滚失败
回滚报错：
提交报错：
详细回滚报错：
[2023-12-05 04:55:45 INFO UpgradeScene 19 1442] 升级版本回滚
[2023-12-05 04:55:45 INFO UpgradeScene 19 1442] 开始执行: su - upgrade_1205 <<EOF
source /home/upgrade_1205/gaussdb.bashrc
gs_upgradectl -t auto-rollback -X /opt/upgrade_1205/1205/pkg/upgrade_1205.xml
EOF
[2023-12-05 04:56:20 INFO UpgradeScene 19 1442] Success: Static configuration matched with old static configuration files.
[GAUSS-51400] : Failed to execute the command: python3 '/data/upgrade_1205/cluster/tool/script/local/UpgradeUtility.py' -t check_guc -U upgrade_1205 --upgrade_bak_path=/data/upgrade_1205/cluster/tmp/binary_upgrade --guc_string="default_transaction_read_only:off" -l /data/upgrade_1205/gaussdb_log/upgrade_1205/om/gs_local.log. Result:{'kwemhisprc10436': 'Success', 'kwepwebenv12243': 'Success', 'kwemhisprc10433': 'Failure'}.
Error:
[SUCCESS] kwemhisprc10436:
[FAILURE] kwemhisprc10433:
No such key to check guc value.
[SUCCESS] kwepwebenv12243:
[GAUSS-51400] : Failed to execute the command: python3 '/data/upgrade_1205/cluster/tool/script/local/UpgradeUtility.py' -t check_guc -U upgrade_1205 --upgrade_bak_path=/data/upgrade_1205/cluster/tmp/binary_upgrade --guc_string="default_transaction_read_only:off" -l /data/upgrade_1205/gaussdb_log/upgrade_1205/om/gs_local.log. Result:{'kwemhisprc10436': 'Success', 'kwepwebenv12243': 'Success', 'kwemhisprc10433': 'Failure'}.
Error:
[SUCCESS] kwemhisprc10436:
[FAILURE] kwemhisprc10433:
No such key to check guc value.
[SUCCESS] kwepwebenv12243:
Last login: Tue Dec 5 04:55:45 CST 2023
详细提交报错：
[2023-12-05 01:56:47 INFO UpgradeScene 19 20402] 提交升级
[2023-12-05 01:56:47 INFO UpgradeScene 19 20402] 开始执行: su - upgrade_1205 <<EOF
source /home/upgrade_1205/gaussdb.bashrc
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1205/1205/pkg/upgrade_1205.xml
EOF
[2023-12-05 01:57:22 INFO UpgradeScene 19 20402] Success: [GAUSS-51400] : Failed to execute the command: python3 '/data/upgrade_1205/cluster/tool/script/local/UpgradeUtility.py' -t check_guc -U upgrade_1205 --upgrade_bak_path=/data/upgrade_1205/cluster/tmp/binary_upgrade --guc_string="default_transaction_read_only:off" -l /data/upgrade_1205/gaussdb_log/upgrade_1205/om/gs_local.log. Result:{'kwemhisprc10436': 'Success', 'kwepwebenv12243': 'Failure', 'kwemhisprc10433': 'Failure'}.
Error:
[SUCCESS] kwemhisprc10436:
[FAILURE] kwemhisprc10433:
No such key to check guc value.
[FAILURE] kwepwebenv12243:
No such key to check guc value.
[GAUSS-51400] : Failed to execute the command: python3 '/data/upgrade_1205/cluster/tool/script/local/UpgradeUtility.py' -t check_guc -U upgrade_1205 --upgrade_bak_path=/data/upgrade_1205/cluster/tmp/binary_upgrade --guc_string="default_transaction_read_only:off" -l /data/upgrade_1205/gaussdb_log/upgrade_1205/om/gs_local.log. Result:{'kwemhisprc10436': 'Success', 'kwepwebenv12243': 'Failure', 'kwemhisprc10433': 'Failure'}.
Error:
[SUCCESS] kwemhisprc10436:
[FAILURE] kwemhisprc10433:
No such key to check guc value.
[FAILURE] kwepwebenv12243:
No such key to check guc value.
Last login: Tue Dec 5 01:56:46 CST 2023
【原因分析】:
【日志信息】(请附上日志文件、截图、coredump信息):
不涉及
【测试代码】:
不涉及
Hey @lixin, Welcome to openGauss Community.
All of the projects in openGauss Community are maintained by @opengauss_bot.
That means the developers can comment below every pull request or issue to trigger Bot Commands.
Please follow instructions at Here to find the details.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
Hi @lixin, please use the command /sig xxx to add a SIG label to this issue.
For example: /sig sqlengine or /sig storageengine or /sig om or /sig ai and so on.
You can find more SIG labels from Here.
If you have no idea about that, please contact with @zhangxubo , @xiangxinyong .
验收日期：2023/12/19
验收版本：gsql (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
验收结论：通过
升级前：
[2023-12-18 15:37:47 INFO UpgradeScene 19 23069] 升级前准备与检查
[2023-12-18 15:37:47 INFO UpgradeScene 19 23069] 1.获取升级前版本
[2023-12-18 15:37:47 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-18 15:37:50 INFO UpgradeScene 19 23069] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
[SUCCESS] kwepwebenv07953:
gsql (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
[SUCCESS] kwepwebenv07954:
gsql (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
灰度升级
[2023-12-18 15:41:10 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc && gs_upgradectl -t auto-upgrade -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml --grey
EOF
[2023-12-18 15:46:17 INFO UpgradeScene 19 23069] Success: Static configuration matched with old static configuration files.
Successfully set upgrade_mode to 0.
Checking upgrade environment.
Successfully checked upgrade environment.
Start to do health check.
Successfully checked cluster status.
Upgrade all nodes.
NOTICE: The directory /data/upgrade_1218/cluster/app_a161c90c will be deleted after commit-upgrade, please make sure there is no personal data.
Performing grey rollback.
No need to rollback.
The directory /data/upgrade_1218/cluster/app_a161c90c will be deleted after commit-upgrade, please make sure there is no personal data.
Installing new binary.
copy certs from /data/upgrade_1218/cluster/app_a161c90c to /data/upgrade_1218/cluster/app_33b035fd.
Successfully copy certs from /data/upgrade_1218/cluster/app_a161c90c to /data/upgrade_1218/cluster/app_33b035fd.
Successfully backup hotpatch config file.
Sync cluster configuration.
Successfully synced cluster configuration.
Switch symbolic link to new binary directory.
Successfully switch symbolic link to new binary directory.
Create CA files for cm beginning.
Create CA files on directory [/data/upgrade_1218/cluster/app_33b035fd/share/sslcert/cm]. file list: ['server.crt', 'cacert.pem', 'client.crt', 'server.key', 'client.key.cipher', 'client.key', 'client.key.rand', 'server.key.rand', 'server.key.cipher']
Start check CMS parameter.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
The nodes ['kwepwebenv06293', 'kwepwebenv07953', 'kwepwebenv07954'] have been successfully upgraded to new version. Then do health check.
Start to do health check.
Successfully checked cluster status.
Waiting for the cluster status to become normal.
.
The cluster status is normal.
Create checkpoint before switching.
Upgrade main process has been finished, user can do some check now.
Once the check done, please execute following command to commit upgrade:
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml
Successfully upgrade all nodes.
Last login: Mon Dec 18 15:41:08 CST 2023
[2023-12-18 15:46:17 INFO UpgradeScene 19 23069] 升级后验证
[2023-12-18 15:46:17 INFO UpgradeScene 19 23069] 1.版本验证
[2023-12-18 15:46:17 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-18 15:46:20 INFO UpgradeScene 19 23069] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
回滚
[2023-12-18 15:46:25 INFO UpgradeScene 19 23069] 开始回滚
[2023-12-18 15:46:25 INFO UpgradeScene 19 23069] 升级版本回滚
[2023-12-18 15:46:25 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_upgradectl -t auto-rollback -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml
EOF
[2023-12-18 15:48:40 INFO UpgradeScene 19 23069] Success: [GAUSS-51400] : Failed to execute the command: sed -i 's/enable_ssl = off/enable_ssl = on/g' /data/upgrade_1218/cluster/cm/cm_agent/cm_agent.conf. Result:{'kwepwebenv07953': 'Failure', 'kwepwebenv07954': 'Failure', 'kwepwebenv06293': 'Failure'}.
Error:
[FAILURE] kwepwebenv06293:
sed: can't read /data/upgrade_1218/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
[FAILURE] kwepwebenv07953:
sed: can't read /data/upgrade_1218/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
[FAILURE] kwepwebenv07954:
sed: can't read /data/upgrade_1218/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
Static configuration matched with old static configuration files.
Performing grey rollback.
Create checkpoint before switching.
Start roll back CM instance.
Switch symbolic link to old binary directory.
Successfully switch symbolic link to old binary directory.
Successfully rollback hotpatch config file.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Switch DN processes for rolling upgrade.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
Rollbacking catalog.
Successfully Rollbacked catalog.
Successfully cleaned new install path.
Rollback succeeded.
Start check CMS parameter.
Last login: Mon Dec 18 15:46:24 CST 2023
[2023-12-18 15:48:40 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 -c "source /home/upgrade_1218/gaussdb.bashrc; gaussdb -V"
[2023-12-18 15:48:40 INFO UpgradeScene 19 23069] Success: gaussdb (openGauss 2.0.4 build a161c90c) compiled at 2022-06-10 17:20:14 commit 0 last mr
升级
[2023-12-18 15:51:36 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc && gs_upgradectl -t auto-upgrade -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml --grey
EOF
[2023-12-18 15:57:04 INFO UpgradeScene 19 23069] Success: Static configuration matched with old static configuration files.
Successfully set upgrade_mode to 0.
Checking upgrade environment.
Successfully checked upgrade environment.
Start to do health check.
Successfully checked cluster status.
Upgrade all nodes.
NOTICE: The directory /data/upgrade_1218/cluster/app_a161c90c will be deleted after commit-upgrade, please make sure there is no personal data.
Performing grey rollback.
No need to rollback.
The directory /data/upgrade_1218/cluster/app_a161c90c will be deleted after commit-upgrade, please make sure there is no personal data.
Installing new binary.
copy certs from /data/upgrade_1218/cluster/app_a161c90c to /data/upgrade_1218/cluster/app_33b035fd.
Successfully copy certs from /data/upgrade_1218/cluster/app_a161c90c to /data/upgrade_1218/cluster/app_33b035fd.
Successfully backup hotpatch config file.
Sync cluster configuration.
Successfully synced cluster configuration.
Switch symbolic link to new binary directory.
Successfully switch symbolic link to new binary directory.
Create CA files for cm beginning.
Create CA files on directory [/data/upgrade_1218/cluster/app_33b035fd/share/sslcert/cm]. file list: ['server.crt', 'cacert.pem', 'client.crt', 'server.key', 'client.key.cipher', 'client.key', 'client.key.rand', 'server.key.rand', 'server.key.cipher']
Start check CMS parameter.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
The nodes ['kwepwebenv06293', 'kwepwebenv07953', 'kwepwebenv07954'] have been successfully upgraded to new version. Then do health check.
Start to do health check.
Successfully checked cluster status.
Waiting for the cluster status to become normal.
.
The cluster status is normal.
Create checkpoint before switching.
Upgrade main process has been finished, user can do some check now.
Once the check done, please execute following command to commit upgrade:
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml
Successfully upgrade all nodes.
Last login: Mon Dec 18 15:51:34 CST 2023
[2023-12-18 15:57:04 INFO UpgradeScene 19 23069] 升级后验证
[2023-12-18 15:57:04 INFO UpgradeScene 19 23069] 1.版本验证
[2023-12-18 15:57:04 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-18 15:57:06 INFO UpgradeScene 19 23069] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
提交
[2023-12-18 15:57:11 INFO UpgradeScene 19 23069] 最后再提交升级
[2023-12-18 15:57:11 INFO UpgradeScene 19 23069] 提交升级
[2023-12-18 15:57:11 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1218/1218/pkg/upgrade_1218.xml
EOF
[2023-12-18 15:58:33 INFO UpgradeScene 19 23069] Success: Wait for the cluster status normal or degrade.
Start check CMS parameter.
Wait for the cluster status normal or degrade.
Start to do health check.
Successfully checked cluster status.
Wait for the cluster status normal or degrade.
Wait for the cluster status normal or degrade.
Start check CMS parameter.
Successfully cleaned old install path.
Commit upgrade succeeded.
Start check CMS parameter.
Wait for the cluster status normal or degrade.
Last login: Mon Dec 18 15:57:11 CST 2023
[2023-12-18 15:58:33 INFO UpgradeScene 19 23069] 版本验证
[2023-12-18 15:58:33 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_ssh -c "gaussdb -V"
EOF
[2023-12-18 15:58:35 INFO UpgradeScene 19 23069] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gaussdb (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
[SUCCESS] kwepwebenv07953:
gaussdb (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
[SUCCESS] kwepwebenv07954:
gaussdb (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
[2023-12-18 15:58:42 INFO UpgradeScene 19 23069] 3.检查数据库状态
[2023-12-18 15:58:42 INFO UpgradeScene 19 23069] 开始执行: su - upgrade_1218 <<EOF
source /home/upgrade_1218/gaussdb.bashrc
gs_om -t status --all
EOF
[2023-12-18 15:58:43 INFO UpgradeScene 19 23069] Success: -----------------------------------------------------------------------
cluster_state : Normal
redistributing : No
balanced : Yes
-----------------------------------------------------------------------
node : 1
node_name : kwepwebenv06293
node : 1
instance_id : 1
node_ip : xx.xx.xxx.xx
data_path : /data/upgrade_1218/cluster/cm/cm_server
type : CMServer
instance_state : Primary
验收日期：2023/12/19
验收版本：gsql (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
验收结论：通过
升级前
[2023-12-19 10:59:52 INFO UpgradeScene 19 10824] 1.获取升级前版本
[2023-12-19 10:59:52 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-19 10:59:54 INFO UpgradeScene 19 10824] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
[SUCCESS] kwepwebenv07953:
gsql (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
[SUCCESS] kwepwebenv07954:
gsql (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
灰度升级
[2023-12-19 11:03:22 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc && gs_upgradectl -t auto-upgrade -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml --grey
EOF
[2023-12-19 11:08:30 INFO UpgradeScene 19 10824] Success: Static configuration matched with old static configuration files.
Successfully set upgrade_mode to 0.
Checking upgrade environment.
Successfully checked upgrade environment.
Start to do health check.
Successfully checked cluster status.
Upgrade all nodes.
NOTICE: The directory /data/upgrade_1219/cluster/app_7f1f151b will be deleted after commit-upgrade, please make sure there is no personal data.
Performing grey rollback.
No need to rollback.
The directory /data/upgrade_1219/cluster/app_7f1f151b will be deleted after commit-upgrade, please make sure there is no personal data.
Installing new binary.
copy certs from /data/upgrade_1219/cluster/app_7f1f151b to /data/upgrade_1219/cluster/app_33b035fd.
Successfully copy certs from /data/upgrade_1219/cluster/app_7f1f151b to /data/upgrade_1219/cluster/app_33b035fd.
Successfully backup hotpatch config file.
Sync cluster configuration.
Successfully synced cluster configuration.
Switch symbolic link to new binary directory.
Successfully switch symbolic link to new binary directory.
Create CA files for cm beginning.
Create CA files on directory [/data/upgrade_1219/cluster/app_33b035fd/share/sslcert/cm]. file list: ['server.crt', 'cacert.pem', 'client.crt', 'server.key', 'client.key.cipher', 'client.key', 'client.key.rand', 'server.key.rand', 'server.key.cipher']
Start check CMS parameter.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
The nodes ['kwepwebenv06293', 'kwepwebenv07953', 'kwepwebenv07954'] have been successfully upgraded to new version. Then do health check.
Start to do health check.
Successfully checked cluster status.
Waiting for the cluster status to become normal.
.
The cluster status is normal.
Create checkpoint before switching.
Upgrade main process has been finished, user can do some check now.
Once the check done, please execute following command to commit upgrade:
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml
Successfully upgrade all nodes.
Last login: Tue Dec 19 11:03:20 CST 2023
[2023-12-19 11:08:30 INFO UpgradeScene 19 10824] 升级后验证
[2023-12-19 11:08:30 INFO UpgradeScene 19 10824] 1.版本验证
[2023-12-19 11:08:30 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-19 11:08:32 INFO UpgradeScene 19 10824] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
回滚
[2023-12-19 11:08:37 INFO UpgradeScene 19 10824] 开始回滚
[2023-12-19 11:08:37 INFO UpgradeScene 19 10824] 升级版本回滚
[2023-12-19 11:08:37 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_upgradectl -t auto-rollback -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml
EOF
[2023-12-19 11:10:54 INFO UpgradeScene 19 10824] Success: [GAUSS-51400] : Failed to execute the command: sed -i 's/enable_ssl = off/enable_ssl = on/g' /data/upgrade_1219/cluster/cm/cm_agent/cm_agent.conf. Result:{'kwepwebenv07953': 'Failure', 'kwepwebenv07954': 'Failure', 'kwepwebenv06293': 'Failure'}.
Error:
[FAILURE] kwepwebenv06293:
sed: can't read /data/upgrade_1219/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
[FAILURE] kwepwebenv07953:
sed: can't read /data/upgrade_1219/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
[FAILURE] kwepwebenv07954:
sed: can't read /data/upgrade_1219/cluster/cm/cm_agent/cm_agent.conf: No such file or directory
Static configuration matched with old static configuration files.
Performing grey rollback.
Create checkpoint before switching.
Start roll back CM instance.
Switch symbolic link to old binary directory.
Successfully switch symbolic link to old binary directory.
Successfully rollback hotpatch config file.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Switch DN processes for rolling upgrade.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
Rollbacking catalog.
Successfully Rollbacked catalog.
Successfully cleaned new install path.
Rollback succeeded.
Start check CMS parameter.
Last login: Tue Dec 19 11:08:37 CST 2023
[2023-12-19 11:10:54 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 -c "source /home/upgrade_1219/gaussdb.bashrc; gaussdb -V"
[2023-12-19 11:10:55 INFO UpgradeScene 19 10824] Success: gaussdb (openGauss 2.0.5 build 7f1f151b) compiled at 2022-09-05 18:59:31 commit 0 last mr
再升级
[2023-12-19 11:13:55 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc && gs_upgradectl -t auto-upgrade -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml --grey
EOF
[2023-12-19 11:19:01 INFO UpgradeScene 19 10824] Success: Static configuration matched with old static configuration files.
Successfully set upgrade_mode to 0.
Checking upgrade environment.
Successfully checked upgrade environment.
Start to do health check.
Successfully checked cluster status.
Upgrade all nodes.
NOTICE: The directory /data/upgrade_1219/cluster/app_7f1f151b will be deleted after commit-upgrade, please make sure there is no personal data.
Performing grey rollback.
No need to rollback.
The directory /data/upgrade_1219/cluster/app_7f1f151b will be deleted after commit-upgrade, please make sure there is no personal data.
Installing new binary.
copy certs from /data/upgrade_1219/cluster/app_7f1f151b to /data/upgrade_1219/cluster/app_33b035fd.
Successfully copy certs from /data/upgrade_1219/cluster/app_7f1f151b to /data/upgrade_1219/cluster/app_33b035fd.
Successfully backup hotpatch config file.
Sync cluster configuration.
Successfully synced cluster configuration.
Switch symbolic link to new binary directory.
Successfully switch symbolic link to new binary directory.
Create CA files for cm beginning.
Create CA files on directory [/data/upgrade_1219/cluster/app_33b035fd/share/sslcert/cm]. file list: ['server.crt', 'cacert.pem', 'client.crt', 'server.key', 'client.key.cipher', 'client.key', 'client.key.rand', 'server.key.rand', 'server.key.cipher']
Start check CMS parameter.
Switching all db processes.
Check cluster state.
Cluster state: [ Cluster State ]
cluster_state : Normal
redistributing : No
current_az : AZ_ALL
Create checkpoint before switching.
Start to wait for om_monitor.
Switching DN processes.
Ready to grey start cluster.
Grey start cluster successfully.
Wait for the cluster status normal or degrade.
Successfully switch all process version
The nodes ['kwepwebenv06293', 'kwepwebenv07953', 'kwepwebenv07954'] have been successfully upgraded to new version. Then do health check.
Start to do health check.
Successfully checked cluster status.
Waiting for the cluster status to become normal.
.
The cluster status is normal.
Create checkpoint before switching.
Upgrade main process has been finished, user can do some check now.
Once the check done, please execute following command to commit upgrade:
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml
Successfully upgrade all nodes.
Last login: Tue Dec 19 11:13:53 CST 2023
[2023-12-19 11:19:01 INFO UpgradeScene 19 10824] 升级后验证
[2023-12-19 11:19:01 INFO UpgradeScene 19 10824] 1.版本验证
[2023-12-19 11:19:01 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_ssh -c "gsql -V"
EOF
[2023-12-19 11:19:03 INFO UpgradeScene 19 10824] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gsql (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
[2023-12-19 11:19:07 INFO UpgradeScene 19 10824] 3.检查数据库状态
[2023-12-19 11:19:07 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_om -t status --all
EOF
[2023-12-19 11:19:08 INFO UpgradeScene 19 10824] Success: -----------------------------------------------------------------------
cluster_state : Normal
redistributing : No
balanced : Yes
-----------------------------------------------------------------------
node : 1
node_name : kwepwebenv06293
node : 1
instance_id : 1
node_ip : xx.xx.xx.xx
data_path : /data/upgrade_1219/cluster/cm/cm_server
type : CMServer
instance_state : Primary
提交
[2023-12-19 11:19:08 INFO UpgradeScene 19 10824] 最后再提交升级
[2023-12-19 11:19:08 INFO UpgradeScene 19 10824] 提交升级
[2023-12-19 11:19:08 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_upgradectl -t commit-upgrade -X /opt/upgrade_1219/1219/pkg/upgrade_1219.xml
EOF
[2023-12-19 11:20:32 INFO UpgradeScene 19 10824] Success: Wait for the cluster status normal or degrade.
Start check CMS parameter.
Wait for the cluster status normal or degrade.
Start to do health check.
Successfully checked cluster status.
Wait for the cluster status normal or degrade.
Wait for the cluster status normal or degrade.
Start check CMS parameter.
Successfully cleaned old install path.
Commit upgrade succeeded.
Start check CMS parameter.
Wait for the cluster status normal or degrade.
Last login: Tue Dec 19 11:19:07 CST 2023
[2023-12-19 11:20:32 INFO UpgradeScene 19 10824] 版本验证
[2023-12-19 11:20:32 INFO UpgradeScene 19 10824] 开始执行: su - upgrade_1219 <<EOF
source /home/upgrade_1219/gaussdb.bashrc
gs_ssh -c "gaussdb -V"
EOF
[2023-12-19 11:20:34 INFO UpgradeScene 19 10824] Success: Successfully execute command on all nodes.
Output:
[SUCCESS] kwepwebenv06293:
gaussdb (openGauss 5.0.1 build 33b035fd) compiled at 2023-12-15 20:19:06 commit 0 last mr
登录 后才可以发表评论