tutorials/experts/source_zh_cn/network/control_flow.ipynb · MindSpore/docs

Create your Gitee Account

Explore and code with more than 12 million developers，Free private repositories ！：）

文件

Clone or Download

control_flow.ipynb 21.35 KB

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 流程控制语句\n",
    "\n",
    "[![在线运行](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_modelarts.png)](https://authoring-modelarts-cnnorth4.huaweicloud.com/console/lab?share-url-b64=aHR0cHM6Ly9taW5kc3BvcmUtd2Vic2l0ZS5vYnMuY24tbm9ydGgtNC5teWh1YXdlaWNsb3VkLmNvbS9ub3RlYm9vay9yMi4wLjAtYWxwaGEvdHV0b3JpYWxzL2V4cGVydHMvemhfY24vbmV0d29yay9taW5kc3BvcmVfY29udHJvbF9mbG93LmlweW5i&imageid=77ef960a-bd26-4de4-9695-5b85a786fb90)&emsp;[![下载Notebook](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_notebook.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.0.0-alpha/tutorials/experts/zh_cn/network/mindspore_control_flow.ipynb)&emsp;\n",
    "[![下载样例代码](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_download_code.png)](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/r2.0.0-alpha/tutorials/experts/zh_cn/network/mindspore_control_flow.py)&emsp;\n",
    "[![查看源文件](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r2.0.0-alpha/tutorials/experts/source_zh_cn/network/control_flow.ipynb)\n",
    "\n",
    "目前主流的深度学习框架的执行模式有两种，分别为静态图模式`GRAPH_MODE`和动态图`PYNATIVE_MODE`模式。\n",
    "\n",
    "在`PYNATIVE_MODE`模式下，MindSpore完全支持Python原生语法的流程控制语句。`GRAPH_MODE`模式下，MindSpore在编译时做了性能优化，因此，在定义网络时使用流程控制语句时会有部分特殊约束，其他部分仍和Python原生语法保持一致。\n",
    "\n",
    "运行模式从动态图切换到静态图时，请留意[静态图语法支持](https://www.mindspore.cn/docs/zh-CN/r2.0.0-alpha/note/static_graph_syntax_support.html#静态图语法支持)。下面我们详细介绍在`GRAPH_MODE`模式下定义网络时流程控制语句的使用方式。\n",
    "\n",
    "## 常量与变量条件\n",
    "\n",
    "在`GRAPH_MODE`模式下定义网络，MindSpore将流程控制语句中的条件表达式分为两类：即常量条件和变量条件。在图编译时可以确定结果为True或False的条件表达式为常量条件，在图编译时不能确定结果为True或False的条件表达式为变量条件。**只有当条件表达式为变量条件时，MindSpore才会在网络中生成控制流算子**。\n",
    "\n",
    "需要注意的是，当网络中存在控制流算子时，网络会被切分成多个执行子图，子图间进行流程跳转和数据传递会产生一定的性能损耗。\n",
    "\n",
    "### 常量条件\n",
    "\n",
    "判断方式：\n",
    "\n",
    "- 条件表达式中不存在Tensor类型，且也不存在元素为Tensor类型的List、Tuple、Dict。\n",
    "- 条件表达式中存在Tensor类型，或者元素为Tensor类型的List、Tuple、Dict，但是表达式结果不受Tensor的值影响。\n",
    "\n",
    "举例：\n",
    "\n",
    "- `for i in range(0,10)`，`i`为标量：潜在的条件表达式`i < 10`在图编译时可以确定结果，因此为常量条件；\n",
    "\n",
    "- `self.flag`，`self.flag`为标量：此处`self.flag`为一个bool类型标量，其值在构建Cell对象时已确定；\n",
    "\n",
    "- `x + 1 < 10`，`x`为标量：此处`x + 1`的值在构建Cell对象时是不确定的，但是在图编译时MindSpore会计算所有标量表达式的结果，因此该表达式的值也是在编译期确定的。\n",
    "\n",
    "- `len(my_list) < 10`，`my_list`为元素是Tensor类型的List对象：该条件表达式包含Tensor，但是表达式结果不受Tensor的值影响，只与`my_list`中Tensor的数量有关；\n",
    "\n",
    "### 变量条件\n",
    "\n",
    "判断方式：\n",
    "\n",
    "- 条件表达式中存在Tensor类型或者元素为Tensor类型的List、Tuple、Dict，并且条件表达式的结果受Tensor的值影响。\n",
    "\n",
    "举例：\n",
    "\n",
    "- `x < y`，`x`和`y`为算子输出。\n",
    "\n",
    "- `x in list`，`x`为算子输出。\n",
    "\n",
    "由于算子输出是图在各个step执行时才能确定，因此上面两个都属于变量条件。\n",
    "\n",
    "## if语句\n",
    "\n",
    "在`GRAPH_MODE`模式下定义网络时，使用`if`语句需要注意：**在条件表达式为变量条件时，在不同分支的同一变量应被赋予相同的数据类型，例如Tensor类型变量要求shape和type一致。**Tensor变量的shape一致性约束详见[ShapeJoin规则](#shapejoin规则)。\n",
    "\n",
    "### 变量条件的if语句\n",
    "\n",
    "在下面代码中，在`if`和`else`分支中，变量`out`在`if`语句不同分支被赋予的Tensor的Shape分别是()和(2,)。网络最终返回的Tensor的shape由条件`x < y`决定，而在图编译时期无法确定`x < y`的结果，因此图编译时期无法确定`out`的Shape是()还是(2,)，MindSpore最终因类型推导失败而抛出异常。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "import numpy as np\n",
    "import mindspore as ms\n",
    "from mindspore import nn\n",
    "\n",
    "class SingleIfNet(nn.Cell):\n",
    "\n",
    "    def construct(self, x, y, z):\n",
    "        # 构造条件表达式为变量条件的if语句\n",
    "        if x < y:\n",
    "            out = x\n",
    "        else:\n",
    "            out = z\n",
    "        out = out + 1\n",
    "        return out\n",
    "\n",
    "forward_net = SingleIfNet()\n",
    "\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "z = ms.Tensor(np.array([1, 2]), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y, z)\n",
    "```\n",
    "\n",
    "执行上面的代码，报错信息如下：\n",
    "\n",
    "```text\n",
    "ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal.\n",
    "Shape Join Failed: shape1 = (), shape2 = (2).\n",
    "```\n",
    "\n",
    "### 常量条件的if语句\n",
    "\n",
    "当`if`语句中的条件表达式为常量条件时，其使用方式与Python原生语法保持一致，并无额外的约束。如下代码中的`if`语句条件表达式`x < y + 1`为常量条件(因为x和y都是标量常量类型)，在图编译时期可确定变量`out`的类型为标量`int`类型，网络可正常编译和执行，输出正确结果`1`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-07T03:35:40.804471Z",
     "start_time": "2022-01-07T03:35:39.226569Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: 1\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import mindspore as ms\n",
    "from mindspore import nn\n",
    "\n",
    "class SingleIfNet(nn.Cell):\n",
    "\n",
    "    def construct(self, z):\n",
    "        x = 0\n",
    "        y = 1\n",
    "\n",
    "        # 构造条件表达式为常量条件的if语句\n",
    "        if x < y + 1:\n",
    "            out = x\n",
    "        else:\n",
    "            out = z\n",
    "        out = out + 1\n",
    "\n",
    "        return out\n",
    "\n",
    "z = ms.Tensor(np.array([0, 1]), dtype=ms.int32)\n",
    "forward_net = SingleIfNet()\n",
    "\n",
    "output = forward_net(z)\n",
    "print(\"output:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## for语句\n",
    "\n",
    "`for`语句会将循环体展开，因此使用`for`语句的网络的子图数量、算子数量取决于`for`语句的循环次数，算子数量过多或者子图过多会消耗更多的硬件资源。\n",
    "\n",
    "下面的示例代码中，`for`语句中的循环体会被执行3次，输出结果为`5`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-07T03:35:40.922574Z",
     "start_time": "2022-01-07T03:35:40.806485Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: 5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "\n",
    "class IfInForNet(nn.Cell):\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        out = 0\n",
    "\n",
    "        # 构造条件表达式为常量条件的for语句\n",
    "        for i in range(0, 3):\n",
    "            # 构造条件表达式为变量条件的if语句\n",
    "            if x + i < y:\n",
    "                out = out + x\n",
    "            else:\n",
    "                out = out + y\n",
    "            out = out + 1\n",
    "\n",
    "        return out\n",
    "\n",
    "forward_net = IfInForNet()\n",
    "\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y)\n",
    "print(\"output:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由于`for`语句会展开循环体内容，所以上面的代码和下面的代码等价："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-07T03:35:40.985953Z",
     "start_time": "2022-01-07T03:35:40.923594Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: 5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "\n",
    "class IfInForNet(nn.Cell):\n",
    "    def construct(self, x, y):\n",
    "        out = 0\n",
    "\n",
    "        # 循环： 0\n",
    "        if x + 0 < y:\n",
    "            out = out + x\n",
    "        else:\n",
    "            out = out + y\n",
    "        out = out + 1\n",
    "        # 循环： 1\n",
    "        if x + 1 < y:\n",
    "            out = out + x\n",
    "        else:\n",
    "            out = out + y\n",
    "        out = out + 1\n",
    "        # 循环： 2\n",
    "        if x + 2 < y:\n",
    "            out = out + x\n",
    "        else:\n",
    "            out = out + y\n",
    "        out = out + 1\n",
    "\n",
    "        return out\n",
    "\n",
    "forward_net = IfInForNet()\n",
    "\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y)\n",
    "print(\"output:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上面两段示例代码我们可以看出，在部分场景下，使用`for`语句会导致出现子图过多的问题时。为了节约硬件资源开销，提升网络编译性能，可尝试将`for`语句等价转换为条件表达式是变量条件的`while`语句。\n",
    "\n",
    "## while语句\n",
    "\n",
    "`while`语句相比`for`语句更为灵活。当`while`的条件为常量时，`while`对循环体的处理和`for`类似，会展开循环体内容。\n",
    "\n",
    "当`while`的条件表达式是变量条件时，`while`语句则不会展开循环体内容，而是在执行图中产生控制流算子，因此可以避免`for`循环带来的子图过多的问题。\n",
    "\n",
    "### 常量条件的while语句\n",
    "\n",
    "下面的示例代码中，`for`语句中的循环体会被执行3次，输出结果为`5`，和上面介绍`for`语句中的示例代码本质上是一样的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-07T03:35:41.051756Z",
     "start_time": "2022-01-07T03:35:40.988065Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: 5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "\n",
    "class IfInWhileNet(nn.Cell):\n",
    "\n",
    "    def construct(self, x, y):\n",
    "        i = 0\n",
    "        out = x\n",
    "        # 构造条件表达式为常量条件的while语句\n",
    "        while i < 3:\n",
    "            # 构造条件表达式为变量条件的if语句\n",
    "            if x + i < y:\n",
    "                out = out + x\n",
    "            else:\n",
    "                out = out + y\n",
    "            out = out + 1\n",
    "            i = i + 1\n",
    "        return out\n",
    "\n",
    "forward_net = IfInWhileNet()\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y)\n",
    "print(\"output:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 变量条件的while语句\n",
    "\n",
    "1. 约束一：**当while语句中的条件表达式是变量条件时，while循环体内部不能出现标量、List、Tuple等非Tensor类型的计算操作**。\n",
    "\n",
    "为了避免产生过多的控制流算子，我们可以尝试使用条件表达式为变量条件的`while`语句重写上面的代码："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "output: 5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "\n",
    "class IfInWhileNet(nn.Cell):\n",
    "\n",
    "    def construct(self, x, y, i):\n",
    "        out = x\n",
    "        # 构造条件表达式为变量条件的while语句\n",
    "        while i < 3:\n",
    "            # 构造条件表达式为变量条件的if语句\n",
    "            if x + i < y:\n",
    "                out = out + x\n",
    "            else:\n",
    "                out = out + y\n",
    "            out = out + 1\n",
    "            i = i + 1\n",
    "        return out\n",
    "\n",
    "forward_net = IfInWhileNet()\n",
    "i = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y, i)\n",
    "print(\"output:\", output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，在上面的代码中，`while`语句的条件表达式为变量条件，`while`循环体不会被展开，`while`循环体内的表达式都是在各个step运行时计算，同时也产生了如下约束：\n",
    "\n",
    "> 当`while`语句中的条件表达式是变量条件时，`while`循环体内部不能出现标量、List、Tuple等非Tensor类型的计算操作。\n",
    "\n",
    "因为这些类型的计算操作是在图编译时期完成的，这与`while`循环体在执行期进行计算的机制是矛盾的。下面我们通过示例代码说明：\n",
    "\n",
    "```Python\n",
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "class IfInWhileNet(nn.Cell):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.nums = [1, 2, 3]\n",
    "\n",
    "    def construct(self, x, y, i):\n",
    "        j = 0\n",
    "        out = x\n",
    "\n",
    "        # 构造条件表达式为变量条件的while语句\n",
    "        while i < 3:\n",
    "            if x + i < y:\n",
    "                out = out + x\n",
    "            else:\n",
    "                out = out + y\n",
    "            out = out + self.nums[j]\n",
    "            i = i + 1\n",
    "            # 在条件表达式为变量条件的while语句循环体内构造标量计算\n",
    "            j = j + 1\n",
    "\n",
    "        return out\n",
    "\n",
    "forward_net = IfInWhileNet()\n",
    "i = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y, i)\n",
    "```\n",
    "\n",
    "上面的代码中，条件表达式`i < 3`为变量条件的`while`循环体内部存在标量计算`j = j + 1`，因此会导致图编译出错。代码在执行时报错信息如下：\n",
    "\n",
    "```text\n",
    "IndexError: mindspore/core/abstract/prim_structures.cc:127 InferTupleOrListGetItem] list_getitem evaluator index should be in range[-3, 3), but got 3.\n",
    "```\n",
    "\n",
    "2. 约束二：**当while语句中的条件表达式是变量条件时，循环体内部不能更改算子的输入shape，并且循环体内与循环体外相同名称变量数据类型应该一致，例如Tensor类型变量要求shape和type一致。**Tensor变量的shape一致性约束详见[ShapeJoin规则](#shapejoin规则)。\n",
    "\n",
    "MindSpore要求网络的同一个算子的输入shape在图编译时是确定的，而在`while`的循环体内部改变算子输入shape的操作是在图执行时生效，两者是矛盾的。\n",
    "\n",
    "下面我们通过示例代码来说明：\n",
    "\n",
    "```Python\n",
    "import numpy as np\n",
    "from mindspore import nn\n",
    "import mindspore as ms\n",
    "from mindspore import ops\n",
    "\n",
    "class IfInWhileNet(nn.Cell):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.expand_dims = ops.ExpandDims()\n",
    "\n",
    "    def construct(self, x, y, i):\n",
    "        out = x\n",
    "        # 构造条件表达式为变量条件的while语句\n",
    "        while i < 3:\n",
    "            if x + i < y:\n",
    "                out = out + x\n",
    "            else:\n",
    "                out = out + y\n",
    "            out = out + 1\n",
    "            # 更改算子的输入shape\n",
    "            out = self.expand_dims(out, -1)\n",
    "            i = i + 1\n",
    "        return out\n",
    "\n",
    "forward_net = IfInWhileNet()\n",
    "i = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "x = ms.Tensor(np.array(0), dtype=ms.int32)\n",
    "y = ms.Tensor(np.array(1), dtype=ms.int32)\n",
    "\n",
    "output = forward_net(x, y, i)\n",
    "```\n",
    "\n",
    "上面的代码中，条件表达式`i < 3`为变量条件的`while`循环体内部的`ExpandDims`算子会改变表达式`out = out + 1`在下一轮循环的输入shape，因此会导致图编译出错。代码在执行时报错信息如下:\n",
    "\n",
    "```text\n",
    "ValueError: mindspore/ccsrc/pipeline/jit/static_analysis/static_analysis.cc:800 ProcessEvalResults] Cannot join the return values of different branches, perhaps you need to make them equal.\n",
    "Shape Join Failed: shape1 = (1), shape2 = (1, 1).\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ShapeJoin规则\n",
    "\n",
    "`unknow_shape`表示动态shape场景下，对应维度的长度是动态的，`unknown_rank`表示动态rank场景下shape的维度是动态的，`shape1`与`shape2`分别表示进行Join的两个分支的shape。当满足下面任一规则时，Shape Join会成功，否则会出现`Shape Join Failed`异常。\n",
    "\n",
    "- 规则1：\n",
    "\n",
    "    shape1和shape2维度均固定且两者维度相等，且shape1[i]等于shape2[i]。\n",
    "- 规则2：\n",
    "\n",
    "    shape1和shape2维度均固定且两者维度相等，且shape1[i]或shape2[i]至少有一个是unknown_shape。\n",
    "- 规则3：\n",
    "\n",
    "    shape1和shape2维度至少有一个是动态的，即shape1或者shape2是动态rank。\n",
    "- 规则4：\n",
    "\n",
    "    shape1和shape2维度固定且两者维度不相等，较小的维度是m, 较大的维度是n。\n",
    "\n",
    "    在0~m-1维范围内满足：\n",
    "\n",
    "    1. shape1[i]或shape2[i]相等。\n",
    "\n",
    "    2. 或shape1[i]与shape2[i]均是unknown_shape。\n",
    "\n",
    "    在m~n-1维范围内满足：维度较大者的shape[i]是unknown_shape。\n",
    "\n",
    "下述列表是Shape Join的规则示例。\n",
    "\n",
    "| shape1 | shape2 | Join结果|\n",
    "| :----- | :----- | :------- |\n",
    "| (3, 4)| (3, 4)| (3, 4) |\n",
    "| (3, 5)| (3, 4)| Join Fail |\n",
    "| (3, 4)| (3, 4, 1)| Join Fail |\n",
    "| (3, unknown_shape) | (3, 4)| (3, unknown_shape) |\n",
    "| unknown_rank | (3, 4)| unknown_rank  |\n",
    "| (3, unknown_shape)| (3, unknown_shape, unknown_shape)| unknown_rank |\n",
    "| (3, unknown_shape)| (4, unknown_shape, unknown_shape)| Join Fail |\n",
    "| (3, unknown_shape)| (3, 4, unknown_shape)| Join Fail |\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

Copy Edit Raw Blame History

authored 2023-11-17 10:40 . modify run online obs links

马建仓 AI 助手

尝试更多

代码解读

代码找茬

代码优化

https://gitee.com/mindspore/docs.git

git@gitee.com:mindspore/docs.git

mindspore

docs

r2.0.0-alpha

MindSpore/docs

About

Releases

Contributors

Activities

MindSpore/docs .gitee-modal { width: 500px !important; }

About

Releases

The Open Source Evaluation Index is derived from the OSS Compass evaluation system, which evaluates projects around the following three dimensions

Contributors

Activities

Search

MindSpore/docs