【测试类型：存储功能】【测试版本：6.0.0】【可靠性】极致RTO，执行tpcc过程中，备机异常PANIC并且core

【标题描述】: 执行tpcc过程中，备机异常panic后stop
【测试类型：SQL功能】【测试版本：6.0.0】 极致RTO，执行tpcc过程中，备机异常panic后stop

【操作系统和硬件信息】(查询命令: cat /etc/system-release, uname -a):
CentOS Linux release 7.6.1908 (Core)

【测试环境】(单机/1主x备x级联备):
一主两备

【被测功能】:
tpcc功能

【测试类型】:
功能测试

【数据库版本】(查询命令: gaussdb -V):
gsql (openGauss 6.0.0 build a734eb27) compiled at 2024-07-31 00:08:54 commit 0 last mr

【预置条件】:
1. 一主两备功能正常
2. enable_default_ustore_table = on
3. tpcc建仓

【操作步骤】(请填写详细的操作步骤):
1. 极致RTO配置
gs_guc set -N all -I all -c "replication_type=1" &&
gs_guc set -N all -I all -c "recovery_parse_workers=4" &&
gs_guc set -N all -I all -c "recovery_redo_workers=4" &&
gs_guc set -N all -I all -c "hot_standby=off" 
gs_om -t stop && gs_om -t start
2. 备机1进行gs_ctl stop
3. 执行tpcc
4. 查询集群状态
gs_om -t status

【预期输出】:
1. 极致RTO配置，成功
2. 备机1进行gs_ctl stop，成功
3. 执行tpcc，成功
4. 查询集群状态，主机、备2状态正常

【实际输出】:
1. 极致RTO配置，成功
2. 备机1进行gs_ctl stop，成功
3. 执行tpcc，成功
4. 查询集群状态，主机正常、备2状态异常

【原因分析】:
1. 这个问题的根因
2. 问题推断过程
3. 还有哪些原因可能造成类似现象
4. 该问题是否有临时规避措施
5. 问题解决方案
6. 预计修复问题时间

【日志信息】(请附上日志文件、截图、coredump信息):
![输入图片说明](https://foruda.gitee.com/images/1723013820939788950/f1fb8c00_8568116.png "屏幕截图")
 **panic，pg日志如下：** 
```
2024-08-05 23:35:38.656 66b0f10e.6129 [unknown] 139704601147136 dn_6003 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  slotname: dn_6002, dummy: 0, restartlsn: 15/AC12720
2024-08-05 23:35:38.656 66b0f10e.6129 [unknown] 139704601147136 dn_6003 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  attempting to remove WAL segments older than log file 000000010000000000000000
2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] PANIC:  row pointer error, offset:7618, flags:1, len:578
2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] CONTEXT:  xlog redo at lsn 1E/DE2775D8, XLOG_UHEAP_UPDATE: TupInfo: oldxid 2, old tupoffset 1, new tupoffset 1, old_tuple_td_id 3, old_tuple_flag 33826. UndoInfo(oldpage): urecptr 70369554890129, blkprev 0, prevurp 0, relOid 119507, partitionOid 0, flag 0, subXid 0, toastLen 0. INPLACE_UPDATE. UndoMetaInfo: zone 4 slot offset 15743352: dbid 525, xid 8952412, lastrecsize 569, allocate 0 switch zone 0.prefixlen 51 suffixlen 4 difflen 512 tHoff 11. td_id 3, reserved 0.
2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] BACKTRACELOG:  tid[114344]'s backtrace:
        /data/relia1022/cluster/app/bin/gaussdb(+0xdbaf6b) [0x565472581f6b]
        /data/relia1022/cluster/app/bin/gaussdb(_Z9errfinishiz+0x59b) [0x56547257968b]
        /data/relia1022/cluster/app/bin/gaussdb(_Z11elog_finishiPKcz+0x213) [0x5654725799b3]
        /data/relia1022/cluster/app/bin/gaussdb(_Z30UHeapXlogUpdateOperatorNewpageP17UpdateRedoBuffersPvbbS1_P14UHeapTupleDatammbbbmP19UpdateRedoAffixLensb+0xc8a) [0x5654731082ba]
        /data/relia1022/cluster/app/bin/gaussdb(+0x1942603) [0x565473109603]
        /data/relia1022/cluster/app/bin/gaussdb(_Z26XLogBlockRedoForExtremeRTOP17XLogRecParseStateP14RedoBufferInfobR12RedoTimeCostS4_+0x266) [0x5654730b50c6]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto18RedoPageWorkerMainEv+0x425) [0x565472f77f95]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto12RedoMainLoopEv+0x1e5) [0x565472f7bdf5]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto22ParallelRedoThreadMainEv+0x2ca) [0x565472f7c0fa]
        /data/relia1022/cluster/app/bin/gaussdb(_Z17GaussDbThreadMainIL15knl_thread_role22EEiP14knl_thread_arg+0x3da) [0x565472b1fd3a]
        /data/relia1022/cluster/app/bin/gaussdb(+0x132b265) [0x565472af2265]
        /lib64/libpthread.so.0(+0x7e65) [0x7f10f4810e65]
        /lib64/libc.so.6(clone+0x6d) [0x7f10f453988d]
        Use addr2line to get pretty function name and line

2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] PANIC:  row pointer error, offset:7618, flags:1, len:578
2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] CONTEXT:  xlog redo at lsn 1E/DE2775D8, XLOG_UHEAP_UPDATE: TupInfo: oldxid 2, old tupoffset 1, new tupoffset 1, old_tuple_td_id 3, old_tuple_flag 33826. UndoInfo(oldpage): urecptr 70369554890129, blkprev 0, prevurp 0, relOid 119507, partitionOid 0, flag 0, subXid 0, toastLen 0. INPLACE_UPDATE. UndoMetaInfo: zone 4 slot offset 15743352: dbid 525, xid 8952412, lastrecsize 569, allocate 0 switch zone 0.prefixlen 51 suffixlen 4 difflen 512 tHoff 11. td_id 3, reserved 0.
2024-08-05 23:36:21.138 66b0f111.6156 [unknown] 139703029790464 dn_6003 0 dn_6001_6002_6003 42809  0 [BACKEND] BACKTRACELOG:  tid[114344]'s backtrace:
        /data/relia1022/cluster/app/bin/gaussdb(+0xdbaf6b) [0x565472581f6b]
        /data/relia1022/cluster/app/bin/gaussdb(_Z9errfinishiz+0x59b) [0x56547257968b]
        /data/relia1022/cluster/app/bin/gaussdb(_Z11elog_finishiPKcz+0x213) [0x5654725799b3]
        /data/relia1022/cluster/app/bin/gaussdb(_Z30UHeapXlogUpdateOperatorNewpageP17UpdateRedoBuffersPvbbS1_P14UHeapTupleDatammbbbmP19UpdateRedoAffixLensb+0xc8a) [0x5654731082ba]
        /data/relia1022/cluster/app/bin/gaussdb(+0x1942603) [0x565473109603]
        /data/relia1022/cluster/app/bin/gaussdb(_Z26XLogBlockRedoForExtremeRTOP17XLogRecParseStateP14RedoBufferInfobR12RedoTimeCostS4_+0x266) [0x5654730b50c6]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto18RedoPageWorkerMainEv+0x425) [0x565472f77f95]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto12RedoMainLoopEv+0x1e5) [0x565472f7bdf5]
        /data/relia1022/cluster/app/bin/gaussdb(_ZN11extreme_rto22ParallelRedoThreadMainEv+0x2ca) [0x565472f7c0fa]
        /data/relia1022/cluster/app/bin/gaussdb(_Z17GaussDbThreadMainIL15knl_thread_role22EEiP14knl_thread_arg+0x3da) [0x565472b1fd3a]
        /data/relia1022/cluster/app/bin/gaussdb(+0x132b265) [0x565472af2265]
        /lib64/libpthread.so.0(+0x7e65) [0x7f10f4810e65]
        /lib64/libc.so.6(clone+0x6d) [0x7f10f453988d]
        Use addr2line to get pretty function name and line
```
core堆栈：

```
(gdb) bt
#0  BBOX_CreateCoredump (file_name=file_name@entry=0x0) at bbox_create.cpp:404
#1  0x00005654727e9a01 in bbox_handler (sig=<optimized out>, si=<optimized out>, uc=<optimized out>) at gs_bbox.cpp:116
#2  bbox_handler (sig=<optimized out>, si=<optimized out>, uc=<optimized out>) at gs_bbox.cpp:103
#3  <signal handler called>
#4  0x00007f10f4471337 in raise () from /lib64/libc.so.6
#5  0x00007f10f4472a28 in abort () from /lib64/libc.so.6
#6  0x000056547257979d in errfinish (dummy=dummy@entry=0) at elog.cpp:797
#7  0x00005654725799b3 in elog_finish (elevel=elevel@entry=22, fmt=fmt@entry=0x5654781e8668 "row pointer error, offset:%u, flags:%u, len:%u") at elog.cpp:1932
#8  0x00005654731082ba in UHeapXlogUpdateOperatorNewpage (buffers=buffers@entry=0x7f0f2573a560, recorddata=recorddata@entry=0x7f0f0bf352ba, inplaceUpdate=inplaceUpdate@entry=true,
    blockInplaceUpdate=blockInplaceUpdate@entry=false, blkdata=<optimized out>, oldtup=oldtup@entry=0x7f0f2573a510, recordlen=566, data_len=519, isinit=false, istoast=false, sameBlock=true,
    recordxid=8952412, affixLens=0x7f0f2573a4f4, isHasCSN=false) at knl_uextremeredo.cpp:921
#9  0x0000565473109603 in UHeapXlogUpdateBlock (blockhead=0x7f0ecd51cf00, blockdatarec=0x7f0ecd51cf50, bufferinfo=0x7f0f2573a710) at knl_uextremeredo.cpp:1390
#10 0x00005654730b50c6 in XLogBlockRedoForExtremeRTO (redoblocktate=redoblocktate@entry=0x7f0ecd51cef8, bufferinfo=bufferinfo@entry=0x7f0f2573a710, notfound=notfound@entry=false,
    readBufCost=..., redoCost=...) at redo_xlogutils.cpp:1858
#11 0x0000565472f77f95 in extreme_rto::RedoPageWorkerMain () at page_redo.cpp:1574
#12 0x0000565472f7bdf5 in extreme_rto::RedoMainLoop () at page_redo.cpp:2545
#13 0x0000565472f7c0fa in extreme_rto::ParallelRedoThreadMain () at page_redo.cpp:2660
#14 0x0000565472b1fd3a in GaussDbAuxiliaryThreadMain<(knl_thread_role)22> (arg=0x7f0fc4d21468) at postmaster.cpp:12007
#15 GaussDbThreadMain<(knl_thread_role)22> (arg=0x7f0fc4d21468) at postmaster.cpp:14236
#16 0x0000565472af2265 in InternalThreadFunc (args=<optimized out>) at postmaster.cpp:14827
#17 0x00007f10f4810e65 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f10f453988d in clone () from /lib64/libc.so.6
```

【测试代码】:

openGauss/openGauss-server

内容风险标识

评论 (3)

openGauss/openGauss-server .gitee-modal { width: 500px !important; }

内容风险标识