Sign in
Sign up
Explore
Enterprise
Education
Search
Help
Terms of use
About Us
Explore
Enterprise
Education
Gitee Premium
Gitee AI
AI teammates
Sign in
Sign up
Fetch the repository succeeded.
description of repo status
Open Source
>
Other
>
Operation System
&&
Donate
Please sign in before you donate.
Cancel
Sign in
Scan WeChat QR to Pay
Cancel
Complete
Prompt
Switch to Alipay.
OK
Cancel
Watch
Unwatch
Watching
Releases Only
Ignoring
128
Star
72
Fork
330
src-openEuler
/
kernel
Closed
Code
Issues
1197
Pull Requests
35
Wiki
Insights
Pipelines
Service
JavaDoc
PHPDoc
Quality Analysis
Jenkins for Gitee
Tencent CloudBase
Tencent Cloud Serverless
悬镜安全
Aliyun SAE
Codeblitz
SBOM
DevLens
Don’t show this again
Update failed. Please try again later!
Remove this flag
Content Risk Flag
This task is identified by
as the content contains sensitive information such as code security bugs, privacy leaks, etc., so it is only accessible to contributors of this repository.
[openEuler release 22.03 LTS]容器批量挂载nas引起内核crash-FS-Cache: Duplicate cookie detected
Done
#I8KL8K
Task
齐特
Opened this issue
2023-11-30 09:53
一、环境基本信息 K8S 1.26.0 openEuler release 22.03 (LTS-SP1) 5.10.0-60.18.0.50.oe2203.x86_64 二、问题现象 1、容器挂载1个nas卷，在服务批量扩容POD后（1扩容到100）会造成node节点节点内核层面挂掉（表现为自动重启后卡死在下图）。  2、容器Ocean平台直接挂载NAS卷如下图  此pod挂载直接底层k8s展现，如下图：  此直接挂载nas操作对应POD所在NODE节点相应的挂载df -h体现如下图：  三、问题现象测试复现 扩容POD挂载nas卷数量多时（pod挂载1个nas卷或多个卷均出现）会发现这个故障现象。经过反复测试挂载nas卷使用的默认nfs3会有这个问题。Nas服务器端开启nfs4协议，客户端配置默认使用Nfs4协议没有这个问题。 四、系统内核dump [15716873.798433] FS-Cache: Duplicate cookie detected [15716873.799237] FS-Cache: Duplicate cookie detected [15716873.799240] FS-Cache: O-cookie c=000000007a36f03b [p=000000006801466d fl=222 nc=0 na=1] [15716873.799241] FS-Cache: O-cookie d=00000000ef4b5a55 n=000000009ec22d36 [15716873.799242] FS-Cache: O-key=[16] '0400000000000000020008010a87dad4' [15716873.799248] FS-Cache: N-cookie c=000000002454c183 [p=000000006801466d fl=2 nc=0 na=1] [15716873.799249] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000182901ca [15716873.799250] FS-Cache: N-key=[16] '0400000000000000020008010a87dad4' [15716873.887990] 00000002000000020008010a87dad4' [15716873.887998] FS-Cache: Duplicate cookie detected [15716873.888000] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=222 nc=0 na=1] [15716873.888001] FS-Cache: O-cookie d=00000000ef4b5a55 n=00000000ebdcf221 [15716873.888002] FS-Cache: O-key=[16] '0400000002000000020008010a87dad4 [15716873.976706] FS-Cache: O-cookie c=000000002120e656 [p=000000006801466d fl=212 nc=0 na=0] [15716874.044485] ' [15716874.044487] FS-Cache: N-cookie c=00000000d729b7f6 [p=000000006801466d fl=2 nc=0 na=1] [15716874.044488] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000dbb335c6 [15716874.044491] FS-Cache: N-key=[16] ' [15716874.153954] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716874.243629] 0400 [15716874.332288] FS-Cache: O-key=[16] '0400 [15716874.439658] 0000 [15716874.529351] 000001000000020008010a87c11b' [15716874.529357] FS-Cache: N-cookie c=00000000ad08b913 [p=000000006801466d fl=2 nc=0 na=1] [15716874.529371] FS-Cache: N-cookie d=00000000ef4b5a55 n=000000006bd28f20 [15716874.618006] 0200 [15716874.685820] FS-Cache: N-key=[16] '040000000100000002000801 [15716874.795274] 000002 [15716874.884967] 0a87c11b [15716874.973609] 0008 [15716875.080975] ' [15716875.080978] FS-Cache: Duplicate cookie detected [15716875.080982] FS-Cache: O-cookie c=00000000d5ed86a5 [p=000000006801466d fl=222 nc=0 na=1] [15716875.170661] 010a87dad4' [15716875.170667] FS-Cache: Duplicate cookie detected [15716875.259312] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716875.327117] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=212 nc=0 na=0] [15716875.327120] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716875.436579] FS-Cache: O-key=[16] ' [15716875.526261] FS-Cache: O-key=[16] '0400000002000000 [15716875.614917] 04 [15716875.722301] 0200 [15716875.811983] 000000010000 [15716875.900633] 08010a87da [15716875.968446] 00020008 [15716876.077893] d4' [15716876.077895] FS-Cache: N-cookie c=0000000034adfe9a [p=000000006801466d fl=2 nc=0 na=1] [15716876.077897] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000caab33dd [15716876.167595] 010a87 [15716876.256239] FS-Cache: N-key=[16] '04000000 [15716876.363595] c11b' [15716876.453285] 02000000020008010a87 [15716876.552858] FS-Cache: N-cookie c=00000000fefb8a66 [p=000000006801466d fl=2 nc=0 na=1] [15716876.552861] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000eeda72d4 [15716876.620671] dad4' [15716876.620674] FS-Cache: Duplicate cookie detected [15716876.620689] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=212 nc=0 na=0] [15716876.730126] FS-Cache: N-key=[16] '040000000100000002000801 [15716876.819804] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716876.819805] FS-Cache: O-key=[16] '0400 [15716876.908449] 0a87c1 [15716877.015851] 000002000000 [15716877.105540] 1b [15716877.194167] 020008010a87dad4' [15716877.194172] FS-Cache: N-cookie c=00000000528aa0c2 [p=000000006801466d fl=2 nc=0 na=1] [15716877.194173] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000cfd9bde5 [15716877.194173] FS-Cache: N-key=[16] '0400000002000000 [15716877.262005] ' [15716877.371436] 020008010a87dad4' [15716898.777870] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [mount.nfs:2788660] [15716902.778041] watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [migration/25:138] [15716902.807561] Modules linked in: [15716902.914915] Modules linked in: [15716903.004595] rpcsec_gss_krb5 [15716903.060987] rpcsec_gss_krb5 [15716903.150658] auth_rpcgss [15716903.209101] auth_rpcgss nfsv4 [15716903.276913] nfsv4 [15716903.344718] ceph libceph [15716903.412525] ceph libceph [15716903.521975] dns_resolver [15716903.622215] dns_resolver nfsv3 [15716903.710853] nfsv3 [15716903.818212] nfs_acl [15716903.907884] nfs_acl nfs [15716903.996518] nfs [15716904.064339] lockd [15716904.173776] lockd [15716904.263448] grace nfs_ssc [15716904.352096] grace [15716904.459455] fscache binfmt_misc [15716904.549133] nfs_ssc [15716904.637761] br_netfilter bridge [15716904.705572] fscache [15716904.815035] stp [15716904.904710] binfmt_misc [15716904.993352] llc [15716905.100712] br_netfilter [15716905.190401] xt_multiport [15716905.279059] bridge stp [15716905.346865] ipt_rpfilter iptable_raw [15716905.456315] llc [15716905.545996] ip_set_hash_net ipip [15716905.634630] xt_multiport ipt_rpfilter [15716905.741999] tunnel4 [15716905.831671] iptable_raw [15716905.920313] ip_tunnel [15716905.988117] ip_set_hash_net ipip [15716906.097550] veth ip6t_REJECT [15716906.187225] tunnel4 [15716906.275855] nf_reject_ipv6 nf_conntrack_netlink [15716906.383215] ip_tunnel [15716906.472885] ipt_REJECT nf_reject_ipv4 [15716906.561522] veth [15716906.670969] xt_addrtype xt_set [15716906.780422] ip6t_REJECT nf_reject_ipv6 [15716906.870097] ip_set_hash_ipportnet [15716906.934786] nf_conntrack_netlink ipt_REJECT [15716907.024457] ip_set_hash_ip ip_set_hash_ipport [15716907.084998] nf_reject_ipv4 xt_addrtype [15716907.192367] ip_set_hash_ipportip ip_set_bitmap_port [15716907.282049] xt_set ip_set_hash_ipportnet [15716907.338414] dummy ip_set [15716907.406123] ip_set_hash_ip ip_set_hash_ipport [15716907.515564] ip_vs_sh ip_vs_wrr [15716907.583376] ip_set_hash_ipportip [15716907.692809] ip_vs_rr ip_vs [15716907.782480] ip_set_bitmap_port dummy [15716907.871111] nf_tables nfnetlink [15716907.978472] ip_set [15716908.068145] ip6table_filter [15716908.156780] ip_vs_sh ip_vs_wrr [15716908.224586] ip6table_nat ip6table_mangle [15716908.334032] ip_vs_rr ip_vs [15716908.423704] ip6_tables xt_MASQUERADE [15716908.512349] nf_tables [15716908.619699] xt_mark iptable_nat [15716908.709387] nfnetlink [15716908.798028] nf_nat xt_comment [15716908.820470] FS-Cache: Duplicate cookie detected [15716908.820473] FS-Cache: O-cookie c=000000008f6cdf98 [p=000000006801466d fl=222 nc=0 na=1] [15716908.820474] FS-Cache: O-cookie d=00000000ef4b5a55 n=00000000a5c8c888 [15716908.820475] FS-Cache: O-key=[16] '0300000000000000020000000a87c11b' [15716908.820481] FS-Cache: N-cookie c=000000003930beb6 [p=000000006801466d fl=2 nc=0 na=1] [15716908.820482] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000dd053649 [15716908.820483] FS-Cache: N-key=[16] '0300000000000000020000000a87c11b' [15716908.905389] ip6table_filter ip6table_nat [15716908.995075] xt_conntrack nf_conntrack [15716909.084748] ip6table_mangle [15716909.139044] nf_defrag_ipv6 nf_defrag_ipv4 [15716909.201668] ip6_tables xt_MASQUERADE [15716909.269476] iptable_filter iptable_mangle [15716909.337294] xt_mark iptable_nat [15716909.446737] ip_tables [15716909.536425] nf_nat [15716909.625066] bonding rfkill [15716909.732430] xt_comment [15716909.822104] overlay [15716909.910738] xt_conntrack [15716909.978542] isofs [15716910.087994] nf_conntrack [15716910.177664] cdrom intel_rapl_msr [15716910.266317] nf_defrag_ipv6 [15716910.373667] intel_rapl_common ipmi_ssif [15716910.463361] nf_defrag_ipv4 [15716910.552004] sunrpc [15716910.619813] iptable_filter [15716910.729259] sb_edac x86_pkg_temp_thermal [15716910.818938] iptable_mangle ip_tables [15716910.907585] intel_powerclamp coretemp [15716911.014946] bonding rfkill [15716911.104621] kvm_intel [15716911.193309] overlay isofs [15716911.302750] xfs iTCO_wdt [15716911.392439] cdrom [15716911.452978] iTCO_vendor_support [15716911.560348] intel_rapl_msr intel_rapl_common [15716911.650029] kvm [15716911.757390] ipmi_ssif sunrpc [15716911.847076] irqbypass [15716911.926346] sb_edac [15716911.982708] i2c_algo_bit [15716912.050512] x86_pkg_temp_thermal [15716912.118335] rapl [15716912.227768] intel_powerclamp [15716912.317435] drm_kms_helper intel_cstate [15716912.406086] coretemp [15716912.513438] pcspkr [15716912.603111] kvm_intel [15716912.701945] bnx2x syscopyarea [15716912.769755] xfs iTCO_wdt [15716912.879192] ses [15716912.968861] iTCO_vendor_support kvm [15716913.056451] intel_uncore sysfillrect [15716913.165890] irqbypass i2c_algo_bit [15716913.273262] cdc_ether enclosure sysimgblt [15716913.362938] rapl drm_kms_helper [15716913.417224] fb_sys_fops [15716913.506795] intel_cstate [15716913.565241] i2c_i801 [15716913.672605] pcspkr [15716913.762284] mei_me usbnet [15716913.841538] bnx2x syscopyarea [15716913.909360] scsi_transport_sas [15716914.018797] ses intel_uncore [15716914.086607] cec [15716914.176307] sysfillrect [15716914.285742] tg3 lpc_ich [15716914.375422] cdc_ether enclosure [15716914.429704] mdio [15716914.500630] sysimgblt fb_sys_fops [15716914.608004] i2c_smbus [15716914.697675] i2c_i801 [15716914.760293] mii [15716914.867650] mei_me usbnet [15716914.957333] mei libcrc32c [15716915.025150] scsi_transport_sas [15716915.134583] sg [15716915.213831] cec [15716915.303504] ipmi_si [15716915.361974] tg3 [15716915.469336] ipmi_devintf [15716915.559013] lpc_ich mdio [15716915.629954] ipmi_msghandler [15716915.732110] i2c_smbus [15716915.833224] acpi_power_meter acpi_pad [15716915.901065] mii mei [15716916.010500] nls_cp437 [15716916.100174] libcrc32c [15716916.188803] vfat [15716916.296161] sg [15716916.385835] fat [15716916.474479] ipmi_si ipmi_devintf [15716916.474489] drm [15716916.474492] ipmi_msghandler [15716916.474494] dm_mod [15716916.474497] acpi_power_meter [15716916.474499] fuse [15716916.474501] acpi_pad [15716916.474504] ext4 [15716916.474507] nls_cp437 [15716916.474509] mbcache [15716916.474512] vfat [15716916.474514] jbd2 [15716916.474517] fat [15716916.474519] sd_mod [15716916.474522] drm [15716916.474524] t10_pi [15716916.474526] dm_mod [15716916.474528] crct10dif_pclmul [15716916.474531] fuse [15716916.474533] crc32_pclmul [15716916.474535] ext4 [15716916.474538] crc32c_intel [15716916.474540] mbcache [15716916.474543] ghash_clmulni_intel [15716916.474545] jbd2 [15716916.474547] megaraid_sas [15716916.474550] sd_mod [15716916.474552] wmi [15716916.474554] t10_pi [15716916.474556] [15716916.474559] crct10dif_pclmul crc32_pclmul [15716916.474563] CPU: 25 PID: 138 Comm: migration/25 Kdump: loaded Not tainted 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.474566] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.474568] crc32c_intel [15716916.474575] RIP: 0010:multi_cpu_stop+0x94/0x100 [15716916.474577] ghash_clmulni_intel megaraid_sas [15716916.474581] Code: ff 4d 00 75 0f 8b 43 20 8b 4b 10 83 c0 01 89 4b 24 89 43 20 e8 4d ec fa ff 41 83 ff 04 74 2e 45 89 fc 4c 89 ef e8 5c ff ff ff <44> 8b 7b 20 45 39 fc 75 aa 41 83 ff 01 76 0a e8 88 e2 01 00 e8 63 [15716916.474584] RSP: 0000:ffffba200688fe70 EFLAGS: 00000202 [15716916.474587] wmi [15716916.474588] [15716916.474590] RAX: ffff91f2bfbb5980 RBX: ffffba200ee97bb0 RCX: 0000000000000000 [15716916.474593] [15716916.474595] RDX: 0000000000000002 RSI: ffffba200ee97c08 RDI: ffffffff9b2165e0 [15716916.474598] RBP: ffffba200ee97bd4 R08: ffff91f2bfba7850 R09: 00000000000001fc [15716916.474600] CPU: 19 PID: 2788660 Comm: mount.nfs Kdump: loaded Not tainted 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.474603] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.474606] R10: ffff91e39941e800 R11: 0000000000000001 R12: 0000000000000001 [15716916.474618] RIP: 0010:fscache_unhash_cookie+0x73/0x80 [fscache] [15716916.474621] R13: ffffffff9b2165e0 R14: 0000000000000000 R15: 0000000000000001 [15716916.474625] Code: 00 00 00 00 ad de 48 89 77 30 48 83 c6 22 48 89 77 38 48 0f ba 34 c5 e0 2c c6 c0 00 c3 f0 48 0f ba 2a 00 73 b4 f3 90 48 8b 0a <83> e1 01 75 f6 eb ec 0f 0b 0f 1f 40 00 0f 1f 44 00 00 41 55 41 54 [15716916.474627] FS: 0000000000000000(0000) GS:ffff91f2bfb80000(0000) knlGS:0000000000000000 [15716916.474629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15716916.474632] RSP: 0018:ffffba2009c73b98 EFLAGS: 00000202 [15716916.474635] CR2: 00007f8ba8e0ce10 CR3: 00000011d018a006 CR4: 00000000001706e0 [15716916.474636] [15716916.474639] RAX: 000000000000658e RBX: ffff91e38aff2f00 RCX: ffff91fbdf56c2b1 [15716916.474641] Call Trace: [15716916.474644] RDX: ffffffffc0c95950 RSI: 0000000000000008 RDI: ffff91fbdf56c280 [15716916.474646] RBP: ffff91fbdf56c280 R08: ffffffff9ba077a0 R09: 0000000000000001 [15716916.474647] R10: ffff91f3045f1800 R11: 0000000000000000 R12: 00000000ffffffff [15716916.474648] R13: ffffffffffffffa3 R14: ffff91f7d49fd5e0 R15: ffff91e389d4a3c0 [15716916.474650] ? stop_machine_yield+0x10/0x10 [15716916.474653] FS: 00007f3be8409840(0000) GS:ffff9202bf880000(0000) knlGS:0000000000000000 [15716916.474656] cpu_stopper_thread+0xaa/0x110 [15716916.474659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15716916.474661] CR2: 000000000065bcd0 CR3: 0000000e7b1be002 CR4: 00000000001706e0 [15716916.474665] smpboot_thread_fn+0xc5/0x160 [15716916.474667] ? sort_range+0x20/0x20 [15716916.474670] Call Trace: [15716916.474674] kthread+0xfe/0x140 [15716916.474678] ? kthread_park+0x90/0x90 [15716916.474681] fscache_cookie_put+0x22/0x90 [fscache] [15716916.474686] ret_from_fork+0x22/0x30 [15716916.474702] nfs_fscache_release_client_cookie+0x2b/0x40 [nfs] [15716916.474714] Kernel panic - not syncing: softlockup: hung tasks [15716916.474721] nfs_free_client+0xe/0x90 [nfs] [15716916.556060] CPU: 25 PID: 138 Comm: migration/25 Kdump: loaded Tainted: G L 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.617647] nfs4_alloc_client+0x90/0x300 [nfsv4] [15716916.763477] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.763479] Call Trace: [15716916.831325] ? nfs_match_client+0x44/0x330 [nfs] [15716916.958428] <IRQ> [15716916.999194] ? dput+0x29/0x1c0 [15716916.999198] ? step_into+0x14d/0x260 [15716917.065991] dump_stack+0x57/0x6a [15716917.065996] panic+0xfb/0x2dc [15716917.101566] nfs_get_client+0x97/0x240 [nfs] [15716917.149601] watchdog_timer_fn.cold+0xc/0x16 [15716917.203885] ? pcpu_block_update_hint_alloc+0xce/0x310 [15716917.254948] __run_hrtimer+0x5e/0x190 [15716917.301981] nfs4_set_client+0xe9/0x140 [nfsv4] [15716917.364453] __hrtimer_run_queues+0x81/0xe0 [15716917.427087] nfs4_init_server+0xe7/0x250 [nfsv4] [15716917.500082] hrtimer_interrupt+0x110/0x2c0 [15716917.555418] nfs4_create_server+0x4e/0xb0 [nfsv4] [15716917.621046] __sysvec_apic_timer_interrupt+0x5f/0xd0 [15716917.682623] nfs4_try_get_tree+0x36/0xb0 [nfsv4] [15716917.749401] asm_call_irq_on_stack+0x12/0x20 [15716917.809924] vfs_get_tree+0x25/0xb0 [15716917.877725] </IRQ> [15716917.948658] do_new_mount+0x152/0x1b0 [15716918.015424] sysvec_apic_timer_interrupt+0x72/0x80 [15716918.078043] __se_sys_mount+0x161/0x1d0 [15716918.131300] asm_sysvec_apic_timer_interrupt+0x12/0x20 [15716918.167901] do_syscall_64+0x33/0x40 [15716918.223224] RIP: 0010:multi_cpu_stop+0x94/0x100 [15716918.292082] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [15716918.349483] Code: ff 4d 00 75 0f 8b 43 20 8b 4b 10 83 c0 01 89 4b 24 89 43 20 e8 4d ec fa ff 41 83 ff 04 74 2e 45 89 fc 4c 89 ef e8 5c ff ff ff <44> 8b 7b 20 45 39 fc 75 aa 41 83 ff 01 76 0a e8 88 e2 01 00 e8 63 [15716918.422488] RIP: 0033:0x7f3be8ab176e [15716918.476769] RSP: 0000:ffffba200688fe70 EFLAGS: 00000202 [15716918.542501] Code: 48 8b 0d ad e6 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 7a e6 0d 00 f7 d8 64 89 01 48 [15716918.614466] RAX: ffff91f2bfbb5980 RBX: ffffba200ee97bb0 RCX: 0000000000000000 [15716918.614469] RDX: 0000000000000002 RSI: ffffba200ee97c08 RDI: ffffffff9b2165e0 [15716918.850832] RSP: 002b:00007ffe4bcda6c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [15716918.905115] RBP: ffffba200ee97bd4 R08: ffff91f2bfba7850 R09: 00000000000001fc [15716918.905118] R10: ffff91e39941e800 R11: 0000000000000001 R12: 0000000000000001 [15716918.979168] RAX: ffffffffffffffda RBX: 00007ffe4bcda850 RCX: 00007f3be8ab176e [15716919.215526] R13: ffffffff9b2165e0 R14: 0000000000000000 R15: 0000000000000001 [15716919.215535] ? stop_machine_yield+0x10/0x10 [15716919.312484] RDX: 00005611bd969b60 RSI: 00005611bd969e10 RDI: 00005611bd969dc0 [15716919.312487] RBP: 00005611bd969ef0 R08: 00005611bd96a130 R09: 0037372e32352e36 [15716919.409450] cpu_stopper_thread+0xaa/0x110 [15716919.511608] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [15716919.608567] smpboot_thread_fn+0xc5/0x160 [15716919.705515] R13: 00005611bd969eb0 R14: 00005611bcf6a66b R15: 0000000000000010 [15716920.468835] ? sort_range+0x20/0x20 [15716920.522081] kthread+0xfe/0x140 [15716920.571163] ? kthread_park+0x90/0x90 [15716920.626486] ret_from_fork+0x22/0x30 [15716920.682120] kexec: Bye!
一、环境基本信息 K8S 1.26.0 openEuler release 22.03 (LTS-SP1) 5.10.0-60.18.0.50.oe2203.x86_64 二、问题现象 1、容器挂载1个nas卷，在服务批量扩容POD后（1扩容到100）会造成node节点节点内核层面挂掉（表现为自动重启后卡死在下图）。  2、容器Ocean平台直接挂载NAS卷如下图  此pod挂载直接底层k8s展现，如下图：  此直接挂载nas操作对应POD所在NODE节点相应的挂载df -h体现如下图：  三、问题现象测试复现 扩容POD挂载nas卷数量多时（pod挂载1个nas卷或多个卷均出现）会发现这个故障现象。经过反复测试挂载nas卷使用的默认nfs3会有这个问题。Nas服务器端开启nfs4协议，客户端配置默认使用Nfs4协议没有这个问题。 四、系统内核dump [15716873.798433] FS-Cache: Duplicate cookie detected [15716873.799237] FS-Cache: Duplicate cookie detected [15716873.799240] FS-Cache: O-cookie c=000000007a36f03b [p=000000006801466d fl=222 nc=0 na=1] [15716873.799241] FS-Cache: O-cookie d=00000000ef4b5a55 n=000000009ec22d36 [15716873.799242] FS-Cache: O-key=[16] '0400000000000000020008010a87dad4' [15716873.799248] FS-Cache: N-cookie c=000000002454c183 [p=000000006801466d fl=2 nc=0 na=1] [15716873.799249] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000182901ca [15716873.799250] FS-Cache: N-key=[16] '0400000000000000020008010a87dad4' [15716873.887990] 00000002000000020008010a87dad4' [15716873.887998] FS-Cache: Duplicate cookie detected [15716873.888000] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=222 nc=0 na=1] [15716873.888001] FS-Cache: O-cookie d=00000000ef4b5a55 n=00000000ebdcf221 [15716873.888002] FS-Cache: O-key=[16] '0400000002000000020008010a87dad4 [15716873.976706] FS-Cache: O-cookie c=000000002120e656 [p=000000006801466d fl=212 nc=0 na=0] [15716874.044485] ' [15716874.044487] FS-Cache: N-cookie c=00000000d729b7f6 [p=000000006801466d fl=2 nc=0 na=1] [15716874.044488] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000dbb335c6 [15716874.044491] FS-Cache: N-key=[16] ' [15716874.153954] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716874.243629] 0400 [15716874.332288] FS-Cache: O-key=[16] '0400 [15716874.439658] 0000 [15716874.529351] 000001000000020008010a87c11b' [15716874.529357] FS-Cache: N-cookie c=00000000ad08b913 [p=000000006801466d fl=2 nc=0 na=1] [15716874.529371] FS-Cache: N-cookie d=00000000ef4b5a55 n=000000006bd28f20 [15716874.618006] 0200 [15716874.685820] FS-Cache: N-key=[16] '040000000100000002000801 [15716874.795274] 000002 [15716874.884967] 0a87c11b [15716874.973609] 0008 [15716875.080975] ' [15716875.080978] FS-Cache: Duplicate cookie detected [15716875.080982] FS-Cache: O-cookie c=00000000d5ed86a5 [p=000000006801466d fl=222 nc=0 na=1] [15716875.170661] 010a87dad4' [15716875.170667] FS-Cache: Duplicate cookie detected [15716875.259312] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716875.327117] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=212 nc=0 na=0] [15716875.327120] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716875.436579] FS-Cache: O-key=[16] ' [15716875.526261] FS-Cache: O-key=[16] '0400000002000000 [15716875.614917] 04 [15716875.722301] 0200 [15716875.811983] 000000010000 [15716875.900633] 08010a87da [15716875.968446] 00020008 [15716876.077893] d4' [15716876.077895] FS-Cache: N-cookie c=0000000034adfe9a [p=000000006801466d fl=2 nc=0 na=1] [15716876.077897] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000caab33dd [15716876.167595] 010a87 [15716876.256239] FS-Cache: N-key=[16] '04000000 [15716876.363595] c11b' [15716876.453285] 02000000020008010a87 [15716876.552858] FS-Cache: N-cookie c=00000000fefb8a66 [p=000000006801466d fl=2 nc=0 na=1] [15716876.552861] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000eeda72d4 [15716876.620671] dad4' [15716876.620674] FS-Cache: Duplicate cookie detected [15716876.620689] FS-Cache: O-cookie c=000000000020df45 [p=000000006801466d fl=212 nc=0 na=0] [15716876.730126] FS-Cache: N-key=[16] '040000000100000002000801 [15716876.819804] FS-Cache: O-cookie d=0000000000000000 n=0000000000000000 [15716876.819805] FS-Cache: O-key=[16] '0400 [15716876.908449] 0a87c1 [15716877.015851] 000002000000 [15716877.105540] 1b [15716877.194167] 020008010a87dad4' [15716877.194172] FS-Cache: N-cookie c=00000000528aa0c2 [p=000000006801466d fl=2 nc=0 na=1] [15716877.194173] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000cfd9bde5 [15716877.194173] FS-Cache: N-key=[16] '0400000002000000 [15716877.262005] ' [15716877.371436] 020008010a87dad4' [15716898.777870] watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [mount.nfs:2788660] [15716902.778041] watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [migration/25:138] [15716902.807561] Modules linked in: [15716902.914915] Modules linked in: [15716903.004595] rpcsec_gss_krb5 [15716903.060987] rpcsec_gss_krb5 [15716903.150658] auth_rpcgss [15716903.209101] auth_rpcgss nfsv4 [15716903.276913] nfsv4 [15716903.344718] ceph libceph [15716903.412525] ceph libceph [15716903.521975] dns_resolver [15716903.622215] dns_resolver nfsv3 [15716903.710853] nfsv3 [15716903.818212] nfs_acl [15716903.907884] nfs_acl nfs [15716903.996518] nfs [15716904.064339] lockd [15716904.173776] lockd [15716904.263448] grace nfs_ssc [15716904.352096] grace [15716904.459455] fscache binfmt_misc [15716904.549133] nfs_ssc [15716904.637761] br_netfilter bridge [15716904.705572] fscache [15716904.815035] stp [15716904.904710] binfmt_misc [15716904.993352] llc [15716905.100712] br_netfilter [15716905.190401] xt_multiport [15716905.279059] bridge stp [15716905.346865] ipt_rpfilter iptable_raw [15716905.456315] llc [15716905.545996] ip_set_hash_net ipip [15716905.634630] xt_multiport ipt_rpfilter [15716905.741999] tunnel4 [15716905.831671] iptable_raw [15716905.920313] ip_tunnel [15716905.988117] ip_set_hash_net ipip [15716906.097550] veth ip6t_REJECT [15716906.187225] tunnel4 [15716906.275855] nf_reject_ipv6 nf_conntrack_netlink [15716906.383215] ip_tunnel [15716906.472885] ipt_REJECT nf_reject_ipv4 [15716906.561522] veth [15716906.670969] xt_addrtype xt_set [15716906.780422] ip6t_REJECT nf_reject_ipv6 [15716906.870097] ip_set_hash_ipportnet [15716906.934786] nf_conntrack_netlink ipt_REJECT [15716907.024457] ip_set_hash_ip ip_set_hash_ipport [15716907.084998] nf_reject_ipv4 xt_addrtype [15716907.192367] ip_set_hash_ipportip ip_set_bitmap_port [15716907.282049] xt_set ip_set_hash_ipportnet [15716907.338414] dummy ip_set [15716907.406123] ip_set_hash_ip ip_set_hash_ipport [15716907.515564] ip_vs_sh ip_vs_wrr [15716907.583376] ip_set_hash_ipportip [15716907.692809] ip_vs_rr ip_vs [15716907.782480] ip_set_bitmap_port dummy [15716907.871111] nf_tables nfnetlink [15716907.978472] ip_set [15716908.068145] ip6table_filter [15716908.156780] ip_vs_sh ip_vs_wrr [15716908.224586] ip6table_nat ip6table_mangle [15716908.334032] ip_vs_rr ip_vs [15716908.423704] ip6_tables xt_MASQUERADE [15716908.512349] nf_tables [15716908.619699] xt_mark iptable_nat [15716908.709387] nfnetlink [15716908.798028] nf_nat xt_comment [15716908.820470] FS-Cache: Duplicate cookie detected [15716908.820473] FS-Cache: O-cookie c=000000008f6cdf98 [p=000000006801466d fl=222 nc=0 na=1] [15716908.820474] FS-Cache: O-cookie d=00000000ef4b5a55 n=00000000a5c8c888 [15716908.820475] FS-Cache: O-key=[16] '0300000000000000020000000a87c11b' [15716908.820481] FS-Cache: N-cookie c=000000003930beb6 [p=000000006801466d fl=2 nc=0 na=1] [15716908.820482] FS-Cache: N-cookie d=00000000ef4b5a55 n=00000000dd053649 [15716908.820483] FS-Cache: N-key=[16] '0300000000000000020000000a87c11b' [15716908.905389] ip6table_filter ip6table_nat [15716908.995075] xt_conntrack nf_conntrack [15716909.084748] ip6table_mangle [15716909.139044] nf_defrag_ipv6 nf_defrag_ipv4 [15716909.201668] ip6_tables xt_MASQUERADE [15716909.269476] iptable_filter iptable_mangle [15716909.337294] xt_mark iptable_nat [15716909.446737] ip_tables [15716909.536425] nf_nat [15716909.625066] bonding rfkill [15716909.732430] xt_comment [15716909.822104] overlay [15716909.910738] xt_conntrack [15716909.978542] isofs [15716910.087994] nf_conntrack [15716910.177664] cdrom intel_rapl_msr [15716910.266317] nf_defrag_ipv6 [15716910.373667] intel_rapl_common ipmi_ssif [15716910.463361] nf_defrag_ipv4 [15716910.552004] sunrpc [15716910.619813] iptable_filter [15716910.729259] sb_edac x86_pkg_temp_thermal [15716910.818938] iptable_mangle ip_tables [15716910.907585] intel_powerclamp coretemp [15716911.014946] bonding rfkill [15716911.104621] kvm_intel [15716911.193309] overlay isofs [15716911.302750] xfs iTCO_wdt [15716911.392439] cdrom [15716911.452978] iTCO_vendor_support [15716911.560348] intel_rapl_msr intel_rapl_common [15716911.650029] kvm [15716911.757390] ipmi_ssif sunrpc [15716911.847076] irqbypass [15716911.926346] sb_edac [15716911.982708] i2c_algo_bit [15716912.050512] x86_pkg_temp_thermal [15716912.118335] rapl [15716912.227768] intel_powerclamp [15716912.317435] drm_kms_helper intel_cstate [15716912.406086] coretemp [15716912.513438] pcspkr [15716912.603111] kvm_intel [15716912.701945] bnx2x syscopyarea [15716912.769755] xfs iTCO_wdt [15716912.879192] ses [15716912.968861] iTCO_vendor_support kvm [15716913.056451] intel_uncore sysfillrect [15716913.165890] irqbypass i2c_algo_bit [15716913.273262] cdc_ether enclosure sysimgblt [15716913.362938] rapl drm_kms_helper [15716913.417224] fb_sys_fops [15716913.506795] intel_cstate [15716913.565241] i2c_i801 [15716913.672605] pcspkr [15716913.762284] mei_me usbnet [15716913.841538] bnx2x syscopyarea [15716913.909360] scsi_transport_sas [15716914.018797] ses intel_uncore [15716914.086607] cec [15716914.176307] sysfillrect [15716914.285742] tg3 lpc_ich [15716914.375422] cdc_ether enclosure [15716914.429704] mdio [15716914.500630] sysimgblt fb_sys_fops [15716914.608004] i2c_smbus [15716914.697675] i2c_i801 [15716914.760293] mii [15716914.867650] mei_me usbnet [15716914.957333] mei libcrc32c [15716915.025150] scsi_transport_sas [15716915.134583] sg [15716915.213831] cec [15716915.303504] ipmi_si [15716915.361974] tg3 [15716915.469336] ipmi_devintf [15716915.559013] lpc_ich mdio [15716915.629954] ipmi_msghandler [15716915.732110] i2c_smbus [15716915.833224] acpi_power_meter acpi_pad [15716915.901065] mii mei [15716916.010500] nls_cp437 [15716916.100174] libcrc32c [15716916.188803] vfat [15716916.296161] sg [15716916.385835] fat [15716916.474479] ipmi_si ipmi_devintf [15716916.474489] drm [15716916.474492] ipmi_msghandler [15716916.474494] dm_mod [15716916.474497] acpi_power_meter [15716916.474499] fuse [15716916.474501] acpi_pad [15716916.474504] ext4 [15716916.474507] nls_cp437 [15716916.474509] mbcache [15716916.474512] vfat [15716916.474514] jbd2 [15716916.474517] fat [15716916.474519] sd_mod [15716916.474522] drm [15716916.474524] t10_pi [15716916.474526] dm_mod [15716916.474528] crct10dif_pclmul [15716916.474531] fuse [15716916.474533] crc32_pclmul [15716916.474535] ext4 [15716916.474538] crc32c_intel [15716916.474540] mbcache [15716916.474543] ghash_clmulni_intel [15716916.474545] jbd2 [15716916.474547] megaraid_sas [15716916.474550] sd_mod [15716916.474552] wmi [15716916.474554] t10_pi [15716916.474556] [15716916.474559] crct10dif_pclmul crc32_pclmul [15716916.474563] CPU: 25 PID: 138 Comm: migration/25 Kdump: loaded Not tainted 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.474566] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.474568] crc32c_intel [15716916.474575] RIP: 0010:multi_cpu_stop+0x94/0x100 [15716916.474577] ghash_clmulni_intel megaraid_sas [15716916.474581] Code: ff 4d 00 75 0f 8b 43 20 8b 4b 10 83 c0 01 89 4b 24 89 43 20 e8 4d ec fa ff 41 83 ff 04 74 2e 45 89 fc 4c 89 ef e8 5c ff ff ff <44> 8b 7b 20 45 39 fc 75 aa 41 83 ff 01 76 0a e8 88 e2 01 00 e8 63 [15716916.474584] RSP: 0000:ffffba200688fe70 EFLAGS: 00000202 [15716916.474587] wmi [15716916.474588] [15716916.474590] RAX: ffff91f2bfbb5980 RBX: ffffba200ee97bb0 RCX: 0000000000000000 [15716916.474593] [15716916.474595] RDX: 0000000000000002 RSI: ffffba200ee97c08 RDI: ffffffff9b2165e0 [15716916.474598] RBP: ffffba200ee97bd4 R08: ffff91f2bfba7850 R09: 00000000000001fc [15716916.474600] CPU: 19 PID: 2788660 Comm: mount.nfs Kdump: loaded Not tainted 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.474603] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.474606] R10: ffff91e39941e800 R11: 0000000000000001 R12: 0000000000000001 [15716916.474618] RIP: 0010:fscache_unhash_cookie+0x73/0x80 [fscache] [15716916.474621] R13: ffffffff9b2165e0 R14: 0000000000000000 R15: 0000000000000001 [15716916.474625] Code: 00 00 00 00 ad de 48 89 77 30 48 83 c6 22 48 89 77 38 48 0f ba 34 c5 e0 2c c6 c0 00 c3 f0 48 0f ba 2a 00 73 b4 f3 90 48 8b 0a <83> e1 01 75 f6 eb ec 0f 0b 0f 1f 40 00 0f 1f 44 00 00 41 55 41 54 [15716916.474627] FS: 0000000000000000(0000) GS:ffff91f2bfb80000(0000) knlGS:0000000000000000 [15716916.474629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15716916.474632] RSP: 0018:ffffba2009c73b98 EFLAGS: 00000202 [15716916.474635] CR2: 00007f8ba8e0ce10 CR3: 00000011d018a006 CR4: 00000000001706e0 [15716916.474636] [15716916.474639] RAX: 000000000000658e RBX: ffff91e38aff2f00 RCX: ffff91fbdf56c2b1 [15716916.474641] Call Trace: [15716916.474644] RDX: ffffffffc0c95950 RSI: 0000000000000008 RDI: ffff91fbdf56c280 [15716916.474646] RBP: ffff91fbdf56c280 R08: ffffffff9ba077a0 R09: 0000000000000001 [15716916.474647] R10: ffff91f3045f1800 R11: 0000000000000000 R12: 00000000ffffffff [15716916.474648] R13: ffffffffffffffa3 R14: ffff91f7d49fd5e0 R15: ffff91e389d4a3c0 [15716916.474650] ? stop_machine_yield+0x10/0x10 [15716916.474653] FS: 00007f3be8409840(0000) GS:ffff9202bf880000(0000) knlGS:0000000000000000 [15716916.474656] cpu_stopper_thread+0xaa/0x110 [15716916.474659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15716916.474661] CR2: 000000000065bcd0 CR3: 0000000e7b1be002 CR4: 00000000001706e0 [15716916.474665] smpboot_thread_fn+0xc5/0x160 [15716916.474667] ? sort_range+0x20/0x20 [15716916.474670] Call Trace: [15716916.474674] kthread+0xfe/0x140 [15716916.474678] ? kthread_park+0x90/0x90 [15716916.474681] fscache_cookie_put+0x22/0x90 [fscache] [15716916.474686] ret_from_fork+0x22/0x30 [15716916.474702] nfs_fscache_release_client_cookie+0x2b/0x40 [nfs] [15716916.474714] Kernel panic - not syncing: softlockup: hung tasks [15716916.474721] nfs_free_client+0xe/0x90 [nfs] [15716916.556060] CPU: 25 PID: 138 Comm: migration/25 Kdump: loaded Tainted: G L 5.10.0-60.18.0.50.oe2203.x86_64 #1 [15716916.617647] nfs4_alloc_client+0x90/0x300 [nfsv4] [15716916.763477] Hardware name: LENOVO System x3650 M5: -[5462R1B]-/00MW307, BIOS -[TCE106KUS-1.10]- 04/17/2015 [15716916.763479] Call Trace: [15716916.831325] ? nfs_match_client+0x44/0x330 [nfs] [15716916.958428] <IRQ> [15716916.999194] ? dput+0x29/0x1c0 [15716916.999198] ? step_into+0x14d/0x260 [15716917.065991] dump_stack+0x57/0x6a [15716917.065996] panic+0xfb/0x2dc [15716917.101566] nfs_get_client+0x97/0x240 [nfs] [15716917.149601] watchdog_timer_fn.cold+0xc/0x16 [15716917.203885] ? pcpu_block_update_hint_alloc+0xce/0x310 [15716917.254948] __run_hrtimer+0x5e/0x190 [15716917.301981] nfs4_set_client+0xe9/0x140 [nfsv4] [15716917.364453] __hrtimer_run_queues+0x81/0xe0 [15716917.427087] nfs4_init_server+0xe7/0x250 [nfsv4] [15716917.500082] hrtimer_interrupt+0x110/0x2c0 [15716917.555418] nfs4_create_server+0x4e/0xb0 [nfsv4] [15716917.621046] __sysvec_apic_timer_interrupt+0x5f/0xd0 [15716917.682623] nfs4_try_get_tree+0x36/0xb0 [nfsv4] [15716917.749401] asm_call_irq_on_stack+0x12/0x20 [15716917.809924] vfs_get_tree+0x25/0xb0 [15716917.877725] </IRQ> [15716917.948658] do_new_mount+0x152/0x1b0 [15716918.015424] sysvec_apic_timer_interrupt+0x72/0x80 [15716918.078043] __se_sys_mount+0x161/0x1d0 [15716918.131300] asm_sysvec_apic_timer_interrupt+0x12/0x20 [15716918.167901] do_syscall_64+0x33/0x40 [15716918.223224] RIP: 0010:multi_cpu_stop+0x94/0x100 [15716918.292082] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [15716918.349483] Code: ff 4d 00 75 0f 8b 43 20 8b 4b 10 83 c0 01 89 4b 24 89 43 20 e8 4d ec fa ff 41 83 ff 04 74 2e 45 89 fc 4c 89 ef e8 5c ff ff ff <44> 8b 7b 20 45 39 fc 75 aa 41 83 ff 01 76 0a e8 88 e2 01 00 e8 63 [15716918.422488] RIP: 0033:0x7f3be8ab176e [15716918.476769] RSP: 0000:ffffba200688fe70 EFLAGS: 00000202 [15716918.542501] Code: 48 8b 0d ad e6 0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 a5 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 7a e6 0d 00 f7 d8 64 89 01 48 [15716918.614466] RAX: ffff91f2bfbb5980 RBX: ffffba200ee97bb0 RCX: 0000000000000000 [15716918.614469] RDX: 0000000000000002 RSI: ffffba200ee97c08 RDI: ffffffff9b2165e0 [15716918.850832] RSP: 002b:00007ffe4bcda6c8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [15716918.905115] RBP: ffffba200ee97bd4 R08: ffff91f2bfba7850 R09: 00000000000001fc [15716918.905118] R10: ffff91e39941e800 R11: 0000000000000001 R12: 0000000000000001 [15716918.979168] RAX: ffffffffffffffda RBX: 00007ffe4bcda850 RCX: 00007f3be8ab176e [15716919.215526] R13: ffffffff9b2165e0 R14: 0000000000000000 R15: 0000000000000001 [15716919.215535] ? stop_machine_yield+0x10/0x10 [15716919.312484] RDX: 00005611bd969b60 RSI: 00005611bd969e10 RDI: 00005611bd969dc0 [15716919.312487] RBP: 00005611bd969ef0 R08: 00005611bd96a130 R09: 0037372e32352e36 [15716919.409450] cpu_stopper_thread+0xaa/0x110 [15716919.511608] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [15716919.608567] smpboot_thread_fn+0xc5/0x160 [15716919.705515] R13: 00005611bd969eb0 R14: 00005611bcf6a66b R15: 0000000000000010 [15716920.468835] ? sort_range+0x20/0x20 [15716920.522081] kthread+0xfe/0x140 [15716920.571163] ? kthread_park+0x90/0x90 [15716920.626486] ret_from_fork+0x22/0x30 [15716920.682120] kexec: Bye!
Comments (
2
)
Sign in
to comment
Status
Done
Backlog
Doing
Done
Declined
Assignees
Not set
Labels
sig/Kernel
help-wanted
Not set
Projects
Unprojected
Unprojected
Pull Requests
None yet
None yet
Successfully merging a pull request will close this issue.
Branches
No related branch
Branches (
-
)
Tags (
-
)
Planed to start - Planed to end
-
Top level
Not Top
Top Level: High
Top Level: Medium
Top Level: Low
Priority
Not specified
Serious
Main
Secondary
Unimportant
Duration
（hours）
参与者（3）
1
https://gitee.com/src-openeuler/kernel.git
git@gitee.com:src-openeuler/kernel.git
src-openeuler
kernel
kernel
Going to Help Center
Search
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大，如何减小？
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release（发行版）
将 PHP 项目自动发布到 packagist.org
Comment
Repository Report
Back to the top
Login prompt
This operation requires login to the code cloud account. Please log in before operating.
Go to login
No account. Register
