多卡训练报错

一、问题现象（附报错日志上下文）：

使用npu进行多卡训练，训练进行第一个step输出loss后报错，但是进程未停止，此代码在gpu上运行正常

warnings.warn(msg, ImportWarning)
/root/miniconda3/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/root/miniconda3/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/root/miniconda3/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
../root/code/MUlFS-CAP-v3/loss/loss.py:140: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:74.)
  corresponding_win_matrix[sw_idx, lw_idx] = 1
/root/code/MUlFS-CAP-v3/loss/loss.py:140: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:74.)
  corresponding_win_matrix[sw_idx, lw_idx] = 1
.. -epoch 0
 -step 0
 -loss_cm 6.637026786804199 -loss_cp 0.0029470184817910194
 -loss_VISDP 0.0022156639024615288 -loss_IRDP 0.0012765157734975219
 -loss_same 0.17338909208774567
*****************************************
[ WARN:0@257.966] global loadsave.cpp:848 imwrite_ Unsupported depth image for selected encoder is fallbacked to CV_8U.
[E compiler_depend.ts:747] [Rank 1] HCCL watchdog thread terminated with exception: 
[ERROR] 2025-03-27-15:41:51 (PID:62782, Device:1, RankID:-1) ERR02005 DIST internal error
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
Traceback (most recent call last):
  File "/root/code/MUlFS-CAP-v3/train.py", line 246, in <module>
    torch.multiprocessing.spawn(train,
  File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
    while not context.join():
  File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 145, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGABRT
[ERROR] 2025-03-27-15:41:53 (PID:62649, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvo5yzzzs'>
  _warnings.warn(warn_message, ResourceWarning)
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpzibvt556'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmplfkdlkwm'>
  _warnings.warn(warn_message, ResourceWarning)
root@autodl-container-acf446bcad-f098e0fb:~/code/MUlFS-CAP-v3# /root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpgth70oub'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmphgi1q8f5'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpsdfm5j4l'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpmz1suwxr'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpz8zuxc75'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmps8x48pn3'>
  _warnings.warn(warn_message, ResourceWarning)
/root/miniconda3/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 102 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  Ascend-cann-kernels-910b_8.0.0_linux-aarch64
--Tensorflow/Pytorch/MindSpore 版本: 2.1.0
--Python 版本 (e.g., Python 3.7.5): Python 3.10.8
--操作系统版本 (e.g., Ubuntu 18.04): Ubuntu 22.04.5 LTS

三、测试步骤：
训练

四、所使用的train.py

```
import os
import time
from pathlib import Path
import math
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as data
import torchvision
from PIL import Image
from torch.utils.data.distributed import DistributedSampler
from torch.nn.parallel import DistributedDataParallel as DDP
from utils import utils
from utils.utils import save_img
from tqdm import tqdm
import args
from loss import loss as Loss
from model import model
import torch.distributed as dist
import torch_npu  # 导入昇腾相关库

def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    # 初始化昇腾分布式训练
    dist.init_process_group("hccl", rank=rank, world_size=world_size)

def cleanup():
    dist.destroy_process_group()

def adjust_learning_rate(optimizer, epoch_count):
    lr = args.args.LR + 0.5 * (args.args.LR_target - args.args.LR) * (
            1 + math.cos((epoch_count - args.args.Warm_epoch) / (args.args.Epoch - args.args.Warm_epoch) * math.pi))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
    return lr

def warmup_learning_rate(optimizer, epoch_count):
    lr = epoch_count * ((args.args.LR_target - args.args.LR) / args.args.Warm_epoch) + args.args.LR
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
    return lr

class TrainDataset(data.Dataset):
    def __init__(self, vis_dir, ir_dir, transform):
        super(TrainDataset, self).__init__()
        self.vis_dir = vis_dir
        self.ir_dir = ir_dir
        self.vis_path, self.vis_paths = self.find_file(self.vis_dir)
        self.ir_path, self.ir_paths = self.find_file(self.ir_dir)
        self.vis_paths = sorted(self.vis_paths)
        self.ir_paths = sorted(self.ir_paths)
        assert (len(self.vis_path) == len(self.ir_path))
        self.transform = transform

def find_file(self, dir):
        path = os.listdir(dir)
        if os.path.isdir(os.path.join(dir, path[0])):
            paths = []
            for dir_name in os.listdir(dir):
                for file_name in os.listdir(os.path.join(dir, dir_name)):
                    paths.append(os.path.join(dir, file_name, file_name))
        else:
            paths = list(Path(dir).glob('*'))
        return path, paths

def read_image(self, path):
        img = Image.open(str(path)).convert('L')
        img = self.transform(img)
        return img

def __getitem__(self, index):
        vis_path = self.vis_paths[index]
        ir_path = self.ir_paths[index]
        vis_img = self.read_image(vis_path)
        ir_img = self.read_image(ir_path)
        return vis_img, ir_img

def __len__(self):
        return len(self.vis_path)

def train(rank, world_size):
    setup(rank, world_size)

device = torch.device(f"npu:{rank}")  # 修改为npu设备

now = int(time.time())
    timeArr = time.localtime(now)
    nowTime = time.strftime("%Y%m%d_%H-%M-%S", timeArr)
    save_model_dir = args.args.train_save_model_dir + "/" + nowTime + "_MulFS-CAP_model"
    save_img_dir = args.args.train_save_img_dir + "/" + nowTime + "_MulFS-CAP_img"
    if rank == 0:
        utils.check_dir(save_model_dir)
        utils.check_dir(save_img_dir)

tf = torchvision.transforms.Compose([
        torchvision.transforms.Resize([args.args.img_size, args.args.img_size]),
        torchvision.transforms.ToTensor()  # (0, 255) -> (0, 1)
    ])

dataset = TrainDataset(args.args.vis_train_dir, args.args.ir_train_dir, tf)
    sampler = DistributedSampler(dataset, num_replicas=world_size, rank=rank)

data_iter = data.DataLoader(
        dataset=dataset,
        shuffle=False,
        batch_size=args.args.batch_size,
        num_workers=4,
        sampler=sampler
    )

iter_num = int(dataset.__len__() / (args.args.batch_size * world_size))
    save_image_iter = int(iter_num / args.args.save_image_num)

Lgrad = Loss.L_Grad().to(device)
    CC = Loss.CorrelationCoefficient().to(device)
    Lcorrespondence = Loss.L_correspondence()

# 使用新的模型类
    model_obj = model.MulFS_CAP_Model().to(device)
    model_obj = DDP(model_obj, device_ids=[rank])

optimizer_FE = torch.optim.Adam([{'params': model_obj.module.base.parameters()},
                                     {'params': model_obj.module.vis_MFE.parameters()},
                                     {'params': model_obj.module.ir_MFE.parameters()},
                                     {'params': model_obj.module.fusion_decoder.parameters()},
                                     {'params': model_obj.module.PAFE.parameters()},
                                     {'params': model_obj.module.decoder.parameters()},
                                     {'params': model_obj.module.MN_vis.parameters()},
                                     {'params': model_obj.module.MN_ir.parameters()}],
                                    lr=0.0002)
    optimizer_VISDP = torch.optim.Adam(model_obj.module.VISDP.parameters(), lr=0.0008)
    optimizer_IRDP = torch.optim.Adam(model_obj.module.IRDP.parameters(), lr=0.0008)
    optimizer_MHCSAvis = torch.optim.Adam(model_obj.module.MHCSA_vis.parameters(), lr=args.args.LR)
    optimizer_MHCSAir = torch.optim.Adam(model_obj.module.MHCSA_ir.parameters(), lr=args.args.LR)
    optimizer_FusionModule = torch.optim.Adam(model_obj.module.fusion_module.parameters(), lr=0.0002)

for epoch in tqdm(range(args.args.Epoch)):
        sampler.set_epoch(epoch)
        if epoch < args.args.Warm_epoch:
            warmup_learning_rate(optimizer_MHCSAvis, epoch)
            warmup_learning_rate(optimizer_MHCSAir, epoch)
        else:
            adjust_learning_rate(optimizer_MHCSAvis, epoch)
            adjust_learning_rate(optimizer_MHCSAir, epoch)

epoch_loss_VISDP = []
        epoch_loss_IRDP = []
        epoch_loss_same = []
        epoch_loss_correspondence_matrix = []
        epoch_loss_correspondence_predict = []

for step, x in enumerate(data_iter):
            vis = x[0].to(device)  # vis
            ir = x[1].to(device)  # ir

with torch.no_grad():
                vis_d, ir_d, _, index_r, _ = model_obj.module.ImageDeformation(vis, ir)

fusion_image, fusion_f, fusion_image_1, fusion_d_image, fusion_d_f, fusion_d_image_1, fusion_image_sample, \
            VISDP_vis_f, VISDP_vis_d_f, IRDP_ir_f, IRDP_ir_d_f, correspondence_matrixs, index_r = model_obj(vis, ir)

# calculate loss
            loss_fusion = Lgrad(vis, ir, fusion_image) + Loss.Loss_intensity(vis, ir, fusion_image) + \
                          Lgrad(vis_d, ir_d, fusion_d_image) + Loss.Loss_intensity(vis_d, ir_d, fusion_d_image)
            loss_fusion_1 = Lgrad(vis, ir, fusion_image_1) + Loss.Loss_intensity(vis, ir, fusion_image_1) + \
                            Lgrad(vis_d, ir_d, fusion_d_image_1) + Loss.Loss_intensity(vis_d, ir_d, fusion_d_image_1)
            loss_0 = loss_fusion
            loss_VISDP = - CC(VISDP_vis_f, fusion_f.detach()) - CC(VISDP_vis_d_f, fusion_d_f.detach())
            loss_IRDP = - CC(IRDP_ir_f, fusion_f.detach()) - CC(IRDP_ir_d_f, fusion_d_f.detach())
            loss_same = F.mse_loss(VISDP_vis_f, IRDP_ir_f) + F.mse_loss(VISDP_vis_d_f, IRDP_ir_d_f)
            loss_1 = 2 * (loss_VISDP + loss_IRDP + loss_same)
            loss_2 = Lgrad(vis, ir, fusion_image_sample) + Loss.Loss_intensity(vis, ir, fusion_image_sample)
            loss_correspondence_matrix, loss_correspondence_matrix_1 = Lcorrespondence(
                correspondence_matrixs, index_r)
            loss_3 = 4 * (loss_correspondence_matrix + loss_correspondence_matrix_1)
            loss = loss_0 + loss_1 + loss_2 + loss_3 + loss_fusion_1

# optimizer network
            optimizer_VISDP.zero_grad()
            optimizer_IRDP.zero_grad()
            optimizer_MHCSAvis.zero_grad()
            optimizer_MHCSAir.zero_grad()
            optimizer_FusionModule.zero_grad()
            optimizer_FE.zero_grad()
            loss.backward()
            optimizer_FE.step()
            optimizer_VISDP.step()
            optimizer_IRDP.step()
            optimizer_MHCSAvis.step()
            optimizer_MHCSAir.step()
            optimizer_FusionModule.step()

epoch_loss_VISDP.append(loss_VISDP.item())
            epoch_loss_IRDP.append(loss_IRDP.item())
            epoch_loss_same.append(loss_same.item())
            epoch_loss_correspondence_matrix.append(loss_correspondence_matrix.item())
            epoch_loss_correspondence_predict.append(loss_correspondence_matrix_1.item())

if rank == 0:
                print(" -epoch " + str(epoch))
                print(" -step " + str(step))
                print(" -loss_cm " + str(loss_correspondence_matrix.item()) + " -loss_cp " + str(
                    loss_correspondence_matrix_1.item()))
                print(" -loss_VISDP " + str(loss_VISDP.item()) + " -loss_IRDP " + str(
                    loss_IRDP.item()))
                print(" -loss_same " + str(loss_same.item()))
                print("*****************************************")

if step % save_image_iter == 0 and rank == 0:
                epoch_step_name = str(epoch) + "epoch" + str(step) + "step"
                if epoch % 2 == 0:
                    output_name = save_img_dir + "/" + epoch_step_name + ".jpg"
                    out = torch.cat([vis, ir_d, fusion_image_1, fusion_image_sample, fusion_d_image_1], dim=2)
                    out = out[0:1, :, :, :]
                    save_img(out, output_name)

if ((epoch + 1) == args.args.Epoch and (step + 1) % iter_num == 0) or (
                    epoch % args.args.save_model_num == 0 and (step + 1) % iter_num == 0) and rank == 0:
                module_name = "MulFS_CAP_Model"
                save_dir = '{:s}/epoch{:d}_iter{:d}_{:s}.pth'.format(save_model_dir, epoch, step + 1,
                                                                     module_name)
                utils.save_state_dir(model_obj.module, save_dir)

if rank == 0:
            epoch_loss_correspondence_matrix_mean = np.mean(epoch_loss_correspondence_matrix)
            epoch_loss_correspondence_predict_mean = np.mean(epoch_loss_correspondence_predict)
            epoch_loss_VISDP_mean = np.mean(epoch_loss_VISDP)
            epoch_loss_IRDP_mean = np.mean(epoch_loss_IRDP)
            epoch_loss_same_mean = np.mean(epoch_loss_same)

print("===========================================================================================")
            print(" -epoch " + str(epoch))
            print(" -loss_cm " + str(epoch_loss_correspondence_matrix_mean) + " -loss_cp " + str(
                epoch_loss_correspondence_predict_mean))
            print(" -loss_VISDP " + str(epoch_loss_VISDP_mean) + " -loss_IRDP " + str(
                epoch_loss_IRDP_mean))
            print(" -loss_same " + str(epoch_loss_same_mean))
            print("===========================================================================================")

cleanup()

if __name__ == "__main__":
    world_size = torch.npu.device_count()  # 修改为npu设备数量
    torch.multiprocessing.spawn(train,
                                args=(world_size,),
                                nprocs=world_size,
                                join=True)
```

Ascend/pytorch

内容风险标识

评论 (2)

Ascend/pytorch .gitee-modal { width: 500px !important; }

内容风险标识