369 Star 3.8K Fork 271

GVPPaddlePaddle / Paddle

 / 详情

请问topk接口如何设置动态变化的k值?

Done
Opened this issue  
2021-03-27 05:50

源自github用户yeyupiaoling:

环境

  • PaddlePaddle 1.4.1
  • Python 3.5
  • Ubuntu 16.04

问题

我项目中需要统计有意义的标签的数据,然后再求topk 的,但是topk的k值好像不能设置变量,如下代码,我该如何操作?

   def cls_ohem2(label):
        # 真实标签对应的概率
        zeros = np.zeros_like(label, dtype=np.float32)
        ones = np.ones_like(label, dtype=np.float32)
        # 统计neg和pos的数量
        valid_inds = np.where(label < zeros, zeros, ones)
        num_valid = np.sum(valid_inds)
        # 选取70%的数据
        num_keep_radio = 0.7
        keep_num = np.array(num_valid * num_keep_radio).astype(np.int32)
        print(keep_num)
        return keep_num

    keep_num = create_tmp_var(name='keep_num', dtype='int32', shape=[1])
    keep_num = fluid.layers.py_func(func=cls_ohem2, x=[label], out=[keep_num])

    keep_num = fluid.layers.cast(keep_num, dtype='float32')

    loss, _ = fluid.layers.topk(input=loss, k=keep_num)
    loss = fluid.layers.reduce_mean(loss)

Comments (11)

Expand operation logs

源自github用户zhaoyuchen2018:
topk 支持variable,你的keep_num被cast为float32的,k应该是int类型的

源自github用户yeyupiaoling:
@zhaoyuchen2018 我这样转换了

     def cls_ohem2(label):
        # 真实标签对应的概率
        zeros = np.zeros_like(label, dtype=np.float32)
        ones = np.ones_like(label, dtype=np.float32)
        # 统计neg和pos的数量
        valid_inds = np.where(label < zeros, zeros, ones)
        num_valid = np.sum(valid_inds)
        # 选取70%的数据
        num_keep_radio = 0.7
        keep_num = np.array(num_valid * num_keep_radio).astype(np.int32)
        print(keep_num)
        return keep_num

    keep_num = create_tmp_var(name='keep_num', dtype='int32', shape=[1])
    keep_num = fluid.layers.py_func(func=cls_ohem2, x=[label], out=[keep_num])

    keep_num = fluid.layers.cast(keep_num, dtype='int64')

    loss, _ = fluid.layers.topk(input=loss, k=keep_num)
    loss = fluid.layers.reduce_mean(loss)

报错,py_func这个接口不知道为啥不能转换,

源自github用户yeyupiaoling:
@zhaoyuchen2018 topk下面这样的操作是正确的吗?可以这样操做吗?

loss = fluid.layers.cross_entropy(input=cls_prob, label=label_filter_invalid)
loss, _ = fluid.layers.topk(input=loss, k=keep_num)

源自github用户yeyupiaoling:
@zhaoyuchen2018 这个是我完整的损失函数

def cls_ohem(cls_prob, label):
    # # 只把pos的label设定为1,其余都为0
    def my_where1(label):
        zeros = np.zeros_like(label, dtype=np.int64)
        label_filter_invalid = np.where(np.less(label, zeros), zeros, label)
        return label_filter_invalid

    label_filter_invalid = create_tmp_var(name='label_filter_invalid', dtype='int64', shape=label.shape)
    label_filter_invalid = fluid.layers.py_func(func=my_where1, x=[label], out=label_filter_invalid)
    loss = fluid.layers.cross_entropy(input=cls_prob, label=label_filter_invalid)

    def cls_ohem2(label):
        # 真实标签对应的概率
        zeros = np.zeros_like(label, dtype=np.float32)
        ones = np.ones_like(label, dtype=np.float32)
        # 统计neg和pos的数量
        valid_inds = np.where(label < zeros, zeros, ones)
        num_valid = np.sum(valid_inds)
        # 选取70%的数据
        num_keep_radio = 0.7
        keep_num = np.array(num_valid * num_keep_radio).astype(np.int32)
        print(keep_num)
        return keep_num

    keep_num = create_tmp_var(name='keep_num', dtype='int32', shape=[1])
    keep_num = fluid.layers.py_func(func=cls_ohem2, x=[label], out=[keep_num])

    keep_num = fluid.layers.cast(keep_num, dtype='int64')

    loss, _ = fluid.layers.topk(input=loss, k=keep_num)
    loss = fluid.layers.reduce_mean(loss)
    return loss

报错信息:

Traceback (most recent call last):
  File "D:/yeyupiaoling/PyCharm/PaddlePaddle-MTCNN/train/train_PNet.py", line 14, in <module>
    image, label, bbox_target, landmark_target, label_cost, bbox_loss, landmark_loss, accuracy, conv4_1, conv4_2, conv4_3 = P_Net()
  File "D:\yeyupiaoling\PyCharm\PaddlePaddle-MTCNN\train\model.py", line 76, in P_Net
    label_cost = cls_ohem(cls_prob=cls_prob, label=label)
  File "D:\yeyupiaoling\PyCharm\PaddlePaddle-MTCNN\train\model.py", line 283, in cls_ohem
    keep_num = fluid.layers.cast(keep_num, dtype='int64')
  File "C:\Users\1596\AppData\Local\Programs\Python\Python35\lib\site-packages\paddle\fluid\layers\tensor.py", line 182, in cast
    attrs={'in_dtype': x.dtype,
AttributeError: 'list' object has no attribute 'dtype'

源自github用户zhaoyuchen2018:
出错的是这个fluid.layers.cast(keep_num, dtype='int64'),cast的参数应该是一个variable,不是一个list

源自github用户yeyupiaoling:
@zhaoyuchen2018 所以我上面也提到这个问题

报错,py_func这个接口不知道为啥不能转换,

那应该如何处理呢?

源自github用户zhaoyuchen2018:
试一下 keep_num=fluid.layers.py_func(func=cls_ohem2, x=label, out=keep_num)

源自github用户yeyupiaoling:
@zhaoyuchen2018 我改成这样了,刚才的错误就没有了,但是程序突然退出了,什么错误都没有输出,应该是windows没有输出。

    keep_num = create_tmp_var(name='keep_num', dtype='int32', shape=[0])
    keep_num = fluid.layers.py_func(func=cls_ohem2, x=label, out=keep_num)

    keep_num = fluid.layers.cast(keep_num, dtype='int32')

    loss, _ = fluid.layers.topk(input=loss, k=keep_num)
    loss = fluid.layers.reduce_mean(loss)

只有这句话

Process finished with exit code -1073740940 (0xC0000374)

源自github用户yeyupiaoling:
@zhaoyuchen2018 我使用Ubuntu试了一下
报错

7fb1f01f8000-7fb1f01f9000 rw-p 00018000 00:00 75324              /lib/x86_64-linux-gnu/libpthread-2.23.so
7fb1f01f9000-7fb1f01fd000 rw-p 00000000 00:00 0
7fb1f0200000-7fb1f0225000 r-xp 00000000 00:00 75323              /lib/x86_64-linux-gnu/ld-2.23.so
7fb1f0225000-7fb1f0226000 r-xp 00025000 00:00 75323              /lib/x86_64-linux-gnu/ld-2.23.so
7fb1f0260000-7fb1f0421000 rw-p 00000000 00:00 0
7fb1f0425000-7fb1f0426000 r--p 00025000 00:00 75323              /lib/x86_64-linux-gnu/ld-2.23.so
7fb1f0426000-7fb1f0427000 rw-p 00026000 00:00 75323              /lib/x86_64-linux-gnu/ld-2.23.so
7fb1f0427000-7fb1f0428000 rw-p 00000000 00:00 0
7fb1f0430000-7fb1f0431000 rwxp 00000000 00:00 0
7fb1f0438000-7fb1f05d0000 r--p 00000000 00:00 57321              /usr/lib/locale/locale-archive
7fb1f05d0000-7fb1f05d1000 rw-p 00000000 00:00 0
7fb1f05e0000-7fb1f05e1000 rw-p 00000000 00:00 0
7fb1f05f0000-7fb1f05f1000 rw-p 00000000 00:00 0
7fb1f0600000-7fb1f0601000 rw-p 00000000 00:00 0
7fb1f0610000-7fb1f0611000 rw-p 00000000 00:00 0
7fb1f0620000-7fb1f0621000 rw-p 00000000 00:00 0
7fb1f0629000-7fb1f0630000 r--s 00000000 00:00 75198              /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7fffcd5b7000-7fffcddb7000 rw-p 00000000 00:00 0                  [stack]
7fffcdf5e000-7fffcdf5f000 r-xp 00000000 00:00 0                  [vdso]
*** Aborted at 1559814099 (unix time) try "date -d @1559814099" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGABRT (@0x65) received by PID 101 (TID 0x7fb1f05e0700) from PID 101; stack trace: ***
    @     0x7fb1efff1390 (unknown)
    @     0x7fb1efc45428 gsignal
    @     0x7fb1efc4702a abort
    @     0x7fb1efc877ea (unknown)
    @     0x7fb1efc9037a (unknown)
    @     0x7fb1efc9453c cfree
    @     0x7fb1d48238bb _ZNK6paddle9operators10TopkKernelINS_8platform8CPUPlaceEfE7ComputeERKNS_9framework16ExecutionContextE._omp_fn.0
    @     0x7fb1d4827a1a paddle::operators::TopkKernel<>::Compute()
    @     0x7fb1d4828103 _ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform8CPUPlaceELb0ELm0EJNS0_9operators10TopkKernelIS8_fEENSA_IS8_dEEEEclEPKcSF_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4_
    @     0x7fb1d5313a96 paddle::framework::OperatorWithKernel::RunImpl()
    @     0x7fb1d5314594 paddle::framework::OperatorWithKernel::RunImpl()
    @     0x7fb1d531276b paddle::framework::OperatorBase::Run()
    @     0x7fb1d42ccf3e paddle::framework::Executor::RunPreparedContext()
    @     0x7fb1d42cff1f paddle::framework::Executor::Run()
    @     0x7fb1d41583ce _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL18pybind11_init_coreERNS_6moduleEEUlRNS2_9framework8ExecutorERKNS6_11ProgramDescEPNS6_5ScopeEibbRKSt6vectorISsSaISsEEE96_vIS8_SB_SD_ibbSI_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNES10_
    @     0x7fb1d41970fe pybind11::cpp_function::dispatcher()
    @           0x4ea137 PyCFunction_Call
    @           0x53c176 PyEval_EvalFrameEx
    @           0x53fc97 (unknown)
    @           0x53bc93 PyEval_EvalFrameEx
    @           0x53fc97 (unknown)
    @           0x53b83f PyEval_EvalFrameEx
    @           0x53fc97 (unknown)
    @           0x5409bf PyEval_EvalCode
    @           0x60cb42 (unknown)
    @           0x60efea PyRun_FileExFlags
    @           0x60f7dc PyRun_SimpleFileExFlags
    @           0x640256 Py_Main
    @           0x4d0001 main
    @     0x7fb1efc30830 __libc_start_main
    @           0x5d6999 _start
    @                0x0 (unknown)
Aborted (core dumped)

源自github用户zhaoyuchen2018:
看coredump是topk计算的时候出错了,可以打印一下topk的输入看看是多少

PaddlePaddle-Coordinator changed issue state from 待办的 to 已完成

Sign in to comment

Status
Assignees
Milestones
Pull Requests
Successfully merging a pull request will close this issue.
Branches
Planed to start   -   Planed to end
-
Top level
Priority
参与者(1)
Python
1
https://gitee.com/paddlepaddle/Paddle.git
git@gitee.com:paddlepaddle/Paddle.git
paddlepaddle
Paddle
Paddle

Search

113223 674803ea 1850385 170725 2838fb2a 1850385