Failed to compile Op [ScaledMaskedSoftmax] for LLaMA 7B-PyTorch inference

python3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.5 --num-gpus 1                     init_kwargs {'torch_dtype': torch.float16}                                                                                                      Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████| 2/2 [00:13<00:00,  6.71s/it]USER: r00625139                                                                                                                                 ASSISTANT:                                                                                                                                      ^[[C                                                                                                                                            E40021: Failed to compile Op [ScaledMaskedSoftmax]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/6.3.RC2/opp/built-in/op_impl/ai_core/tbe/impl/scaled_masked_softmax.py failed with errormsg/stack: File "/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/impl/scaled_masked_softmax.py", line 685, in scaled_masked_softmax                                                                                         f'should be within the range of [32, 2048] and be divisible by 32. Currently, it is {w_dim}.')                                              RuntimeError: Error! The last dimension of the input tensor should be within the range of [32, 2048] and be divisible by 32. Currently, it is 48.                                                                                                                                               ], optype: [ScaledMaskedSoftmax])                                                                                                                       Solution: See the host log for details, and then check the Python stack where the error log is reported.                                        TraceBack (most recent call last):                                                                                                              Compile op[ScaledMaskedSoftmax] failed, oppath[/usr/local/Ascend/ascend-toolkit/6.3.RC2/opp/built-in/op_impl/ai_core/tbe/impl/scaled_masked_softmax.py], optype[ScaledMaskedSoftmax], taskID[70]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:890]                                                                                                                                         [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281459583599088] recompile single op[ScaledMaskedSoftmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:969]                                                                                                   [SubGraphOpt][Compile][ParalCompOp] Thread[281459583599088] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1015]                                                                                                                                           [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1131]                                        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1356]                                                                                                                                   Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph2[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]                                                                                                                           subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:964]                                              build graph failed, graph id:37, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1511]                                                       [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]                                                                                                                                         [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]                                                          build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]                                                                                                                                                                                             terminate called after throwing an instance of 'std::runtime_error'                                                                               what():  ASCEND kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.            For getting the stacktrace of OP in PyTorch, consider passing TASK_QUEUE_ENABLE=0.                                                              Aborted (core dumped)

Ascend/modelzoo
暂停

内容风险标识

评论 (9)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识