如图,多任务训练场景,期望效果如下,pytorch中的实现为detath:
代码片段:
mlm_loss = generate_tensor()
mlm_start = 0
mlm_end = self.mlm_batch_size * seq_length
if self.mlm_batch_size > 0:
per_mlm_loss = self.slice(per_example_loss, (mlm_start,), (mlm_end,), (1,))
mlm_weights = self.slice(label_weights, (mlm_start,), (mlm_end,), (1,))
mlm_loss = self.reduce_sum_1dim(per_mlm_loss, ()) / self.reduce_sum_1dim(mlm_weights, ())
F.stop_gradient(per_mlm_loss)
实际效果如下,加入stop_gradient的算子仍然计算了反向:
Please add labels (comp or sig), also you can visit "https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md" to find more.
为了让问题更快得到响应,请您为该issue打上**组件(comp)或兴趣组(sig)**标签,打上标签的问题可以直接推送给责任人进行处理。更多的标签可以查看 https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md"
以组件问题为例,如果你发现问题是data组件造成的,你可以这样评论:
//comp/data
当然你也可以向data SIG组求助,可以这样写:
//comp/data
//sig/data
如果是一个简单的问题,你可以留给刚进入社区的小伙伴来回答,这时候你可以这样写:
//good-first-issue
恭喜你,你已经学会了使用命令来打标签,接下来就在下面的评论里打上标签吧!
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。
//comp/operator
@张清华 @liangchenghui 请帮忙看一下这个问题
/assign @chenfei_mindspore
/assign @YuJianfeng 贴一下之前的分析结论,然后走给并行相关责任人吧
@steve stop_gradient在图上是一个算子,不能被悬空,请参照如下用法
class Net(nn.Cell):
def __init__(self):
super(Net, self).__init__()
self.matmul = ops.MatMul()
def construct(self, x, y):
out1 = self.matmul(x, y)
out2 = self.matmul(x, y)
out2 = stop_gradient(out2)
out = out1 + out2
return out
/assign @steve
@chenfei52: Gitee didn't allow you to assign to: steve9.
Choose one of following members as assignee.
In response to this:
/assign @steve
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the opensourceways/test-infra repository.
验证修改后报错如下:
[ERROR] PARALLEL(167860,python3):2021-06-22-10:07:15.340.501 [mindspore/ccsrc/frontend/parallel/step_parallel.cc:2457] GetLossNodeGradOutputLayout] : The pointer[prim] is null.
Sign in to comment