【测试类型：故障注入】【测试版本：6.0.0】【可靠性】极致RTO场景，tpcc执行过程中，备机1注入kill -19故障10s后恢复，tpcc执行偶现异常，主机异常stop

【标题描述】: 极致RTO场景，tpcc执行过程中，备机1注入kill -19故障10s后恢复，tpcc执行异常
【测试类型：故障注入】【测试版本：6.0.0】【可靠性】极致RTO场景，tpcc执行过程中，备机1注入kill -19故障10s后恢复，tpcc执行偶现异常，主机异常stop

【操作系统和硬件信息】(查询命令: cat /etc/system-release, uname -a):
CentOS Linux release 7.7.1908 (Core)

【测试环境】(单机/1主x备x级联备):
一主两备

【被测功能】:
ustore极致RTO

【测试类型】:
可靠性

【数据库版本】(查询命令: gaussdb -V):
gaussdb (openGauss 6.0.0 build a734eb27) compiled at 2024-07-31 00:08:54 commit 0 last mr

【预置条件】:
1. 默认表类型为ustore
enable_default_ustore_table= on

【操作步骤】(请填写详细的操作步骤):
1. 开启极致RTO
gs_guc set -N all -I all -c "replication_type=1";
gs_guc set -N all -I all -c "recovery_parse_workers=4";
gs_guc set -N all -I all -c "recovery_redo_workers=4";
gs_guc set -N all -I all -c "hot_standby=off";
gs_om -t stop
gs_om -t start
2. 运行TPCC
source /etc/profile; cd /autotest_software/benchmark/run; . runBenchmark.sh props.pg
3. 步骤3过程中，备机1进行kill -19
ps ux | grep -w 'M standby' | grep -v grep
kill -19 29716;
4. 等待10S，恢复备机数据库进程kill - 18
kill -18 29716;
5. 查看主机tpcc执行结果

【预期输出】:
1. 开启极致RTO，成功
2. 运行TPCC，开始执行
3. 步骤3过程中，备机1进行kill -19，成功
4. 等待10S，恢复备机数据库进程kill - 18，成功
5. 查看主机tpcc执行结果，正常

【实际输出】:
1. 开启极致RTO，成功
2. 运行TPCC，开始执行
3. 步骤3过程中，备机1进行kill -19，成功
4. 等待10S，恢复备机数据库进程kill - 18，成功
5. 查看主机tpcc执行结果，异常
![输入图片说明](https://foruda.gitee.com/images/1722825416273663695/e4cbeb54_8568116.png "屏幕截图")
【原因分析】:
1. 极致RTO，TPCC执行异常

【日志信息】(请附上日志文件、截图、coredump信息):
 **tpcc执行异常日志：**

```
**H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H19:14:26,824 [Thread-19] ERROR  jTPCCTData : Unexpected SQLException in PAYMENT
19:14:26,827 [Thread-19] ERROR  jTPCCTData : ERROR: [UPAGE_VERIFY|TUPLE]corrupted tuple: tupsize = 379214832, rpsize = 616.
org.postgresql.util.PSQLException: ERROR: [UPAGE_VERIFY|TUPLE]corrupted tuple: tupsize = 379214832, rpsize = 616.
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2794)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2533)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:314)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:449)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:373)
        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:146)
        at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:123)
        at jTPCCTData.executePayment(jTPCCTData.java:916)
        at jTPCCTData.execute(jTPCCTData.java:99)
        at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:160)
        at jTPCCTerminal.run(jTPCCTerminal.java:88)
        at java.lang.Thread.run(Thread.java:748)
** 
```

**pg日志：详细日志见附件**

```

2024-08-02 19:14:11.901 66acbf74.5047 tpccdb 140234415929088 PostgreSQL JDBC Driver 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  [ApplyUndoActions:] Error occured while executing undo actions TransactionId: 7009659, latest_urp: 316659547887062, dbid: 16384
2024-08-02 19:14:11.901 66acbf74.5047 tpccdb 140234415929088 PostgreSQL JDBC Driver 0 dn_6001_6002_6003 XX001  0 [BACKEND] ERROR:  [UPAGE_VERIFY|TUPLE] tup xid inconsistency with td: tupxid = 5593886, tdxid = 5797958, urp 316659530343907.
2024-08-02 19:14:11.901 66acbf74.5047 tpccdb 140234415929088 PostgreSQL JDBC Driver 0 dn_6001_6002_6003 XX001  0 [BACKEND] STATEMENT:  ROLLBACK
2024-08-02 19:14:11.901 66acbf74.5047 tpccdb 140234415929088 PostgreSQL JDBC Driver 0 dn_6001_6002_6003 XX001  0 [BACKEND] BACKTRACELOG:  tid[120701]'s backtrace:
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(+0xdbaf6b) [0x56525bfe6f6b]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z9errfinishiz+0x59b) [0x56525bfde68b]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z11UpageVerifyP19UHeapPageHeaderDatamP9tupleDescP12RelationDatabht+0xd2) [0x56525cb41f02]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z16UHeapUndoActionsP10URecVectoriimjjjbiPjS1_+0x89a) [0x56525cb5dffa]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z22VerifyAndDoUndoActionsmmmbb+0x11e) [0x56525cb5c60e]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z18ExecuteUndoActionsmmmmb15UndoPersistence+0x2c) [0x56525cb5c85c]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z21TryExecuteUndoActionsP20TransactionStateData15UndoPersistence+0x276) [0x56525ca32c16]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z16ApplyUndoActionsv+0x287) [0x56525ca33537]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z24CommitTransactionCommandb+0x11d) [0x56525ca358cd]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(+0x13ca12e) [0x56525c5f612e]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(+0x13d255d) [0x56525c5fe55d]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z12PostgresMainiPPcPKcS2_+0x7b59) [0x56525c60c979]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(+0x132b1d5) [0x56525c5571d5]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(_Z17GaussDbThreadMainIL15knl_thread_role1EEiP14knl_thread_arg+0x338) [0x56525c5874c8]
        /data/peilq_issue_app/peilq_issue/cluster/app/bin/gaussdb(+0x132b265) [0x56525c557265]
        /lib64/libpthread.so.0(+0x7e65) [0x7f8c935f9e65]
        /lib64/libc.so.6(clone+0x6d) [0x7f8c9332288d]
        Use addr2line to get pretty function name and line

```

【测试代码】:

openGauss/openGauss-server

内容风险标识

评论 (6)

openGauss/openGauss-server .gitee-modal { width: 500px !important; }

内容风险标识