docs/mindspore/source_zh_cn/design/auto_gradient.ipynb · MindSpore/docs

加入 Gitee

与超过 1200万开发者一起发现、参与优秀开源项目，私有仓库也完全免费：）

该仓库未声明开源许可证文件（LICENSE），使用请关注具体项目描述及其代码上游依赖。

克隆/下载

auto_gradient.ipynb 15.19 KB

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 函数式微分编程\n",
    "\n",
    "[![](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/resource/_static/logo_source.png)](https://gitee.com/mindspore/docs/blob/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/auto_gradient.ipynb)\n",
    "\n",
    "## 自动微分\n",
    "\n",
    "深度学习等现代AI算法，通过使用大量的数据来学习拟合出一个优化后带参模型，其中使用的学习算法，多是基于现实数据自模型中的经验误差来反向传播以更新模型的参数，**自动微分技术（Automatic Differentiation，AD）**正是其中的关键技术。\n",
    "\n",
    "自动微分是一种介于数值微分与符号微分之间的一种求导方法。自动微分的核心思想是将计算机程序中的运算操作分解为一个有限的基本操作合集，且合集中基本操作的求导规则均为已知的。在完成每一个基本操作的求导后，使用链式求导法则将结果组合得到整体程序的求导结果。\n",
    "\n",
    "链式求导法则：\n",
    "\n",
    "$$\n",
    "(f\\circ g)^{'}(x)=f^{'}(g(x))g^{'}(x) \\tag{1}\n",
    "$$\n",
    "\n",
    "根据对分解后的基本操作求导和链式规则的组合不同，自动微分可以分为**前向模式**和**反向模式**。\n",
    "\n",
    "- 前向自动微分（Forward Automatic Differentiation，也叫做 tangent linear mode AD）或者前向累积梯度（前向模式）。\n",
    "\n",
    "- 后向自动微分（Reverse Automatic Differentiation，也叫做 adjoint mode AD）或者说反向累计梯度（反向模式）。\n",
    "\n",
    "我们以公式 (2) 为例介绍前向微分与反向微分的具体计算方式：\n",
    "\n",
    "$$\n",
    "y=f(x_{1},x_{2})=ln(x_{1})+x_{1}x_{2}-sin(x_{2}) \\tag{2}\n",
    "$$\n",
    "\n",
    "当我们使用前向自动微分求公式 (2) 在$x_{1}=2,x_{2}=5$处的导数 $\\frac{\\partial y}{\\partial x_{1}}$ 时，前向自动微分的求导方向与原函数的求值方向一致，原函数结果与微分结果可以被同时获得。\n",
    "\n",
    "![forward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/images/auto_gradient_forward.png)\n",
    "\n",
    "当使用反向自动微分时，反向自动微分的求导方向与原函数的求值方向相反，微分结果需要依赖原函数的运行结果。\n",
    "\n",
    "![backward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/images/auto_gradient_backward.png)\n",
    "\n",
    "MindSpore先构建的是基于反向模式的自动微分，并在该方法的基础上实现了正向微分。\n",
    "\n",
    "为了进一步说明前向微分与反向微分的区别，我们将被求导的原函数，泛化为具有N输入与M输出的函数F：\n",
    "\n",
    "$$\n",
    "(Y_{1},Y_{2},...,Y_{M})=F(X_{1},X_{2},...,X_{N}) \\tag{3}\n",
    "$$\n",
    "\n",
    "函数 $F()$ 的导数本身为一个雅可比矩阵(Jacobian matrix)。\n",
    "\n",
    "$$\n",
    "\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial Y_{1}}{\\partial X_{1}}& ... & \\frac{\\partial Y_{1}}{\\partial X_{N}} \\\\\n",
    "   ... & ... & ... \\\\\n",
    "   \\frac{\\partial Y_{M}}{\\partial X_{1}} & ... & \\frac{\\partial Y_{M}}{\\partial X_{N}}\n",
    "  \\end{matrix}\n",
    "  \\right]\n",
    "\\tag{4}\n",
    "$$\n",
    "\n",
    "### 前向自动微分\n",
    "\n",
    "在前向自动微分当中，我们是从输入开始向输出的方向计算的，因此每一次计算我们可以求得输出对某一输入的导数，即雅可比矩阵中的一列。\n",
    "\n",
    "$$\n",
    "\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial Y_{1}}{\\partial X_{1}}\\\\\n",
    "   ...  \\\\\n",
    "   \\frac{\\partial Y_{M}}{\\partial X_{1}}\n",
    "  \\end{matrix}\n",
    "  \\right]\n",
    "\\tag{5}\n",
    "$$\n",
    "\n",
    "为了求取该列的值，自动微分将程序分解为一系列求导规则已知的基本操作，这些基本操作也可以被泛化表达为具有$n$输入和$m$输出的函数$f$：\n",
    "\n",
    "$$\n",
    "(y_{1},y_{2},...,y_{m})=f(x_{1},x_{2},...,x_{n}) \\tag{6}\n",
    "$$\n",
    "\n",
    "由于我们的已知基础函数 $f$ 的求导规则，即 $f$ 的雅可比矩阵是已知的。于是我们可以对$f$计算雅可比向量积（Jvp, Jacobian-vector-product），并应用链式求导法则获得导数结果。\n",
    "\n",
    "$$\n",
    "\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial y_{1}}{\\partial X_{i}}\\\\\n",
    "   ...  \\\\\n",
    "   \\frac{\\partial y_{m}}{\\partial X_{i}}\n",
    "  \\end{matrix}\n",
    "  \\right]=\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}}& ... & \\frac{\\partial y_{1}}{\\partial x_{n}} \\\\\n",
    "   ... & ... & ... \\\\\n",
    "   \\frac{\\partial y_{m}}{\\partial x_{1}} & ... & \\frac{\\partial y_{M}}{\\partial x_{n}}\n",
    "  \\end{matrix}\n",
    "  \\right]\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial x_{1}}{\\partial X_{i}}\\\\\n",
    "   ...  \\\\\n",
    "   \\frac{\\partial x_{n}}{\\partial X_{i}}\n",
    "  \\end{matrix}\n",
    "  \\right]\n",
    "\\tag{7}\n",
    "$$\n",
    "\n",
    "### 反向自动微分\n",
    "\n",
    "在反向自动微分当中，我们是从输出开始向输入的方向计算的，因此每一次计算我们可以求得某一输出对输入的导数，即雅可比矩阵中的一行。\n",
    "\n",
    "$$\n",
    "\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial Y_{1}}{\\partial X_{1}}& ... & \\frac{\\partial Y_{1}}{\\partial X_{N}} \\\\\n",
    "  \\end{matrix}\n",
    "  \\right]\n",
    "\\tag{8}\n",
    "$$\n",
    "\n",
    "为了求取该列的值， 自动微分将程序分解为一系列求导规则已知的基本操作，这些基本操作也可以被泛化表达为具有n输入和m输出的函数$f$：\n",
    "\n",
    "$$\n",
    "(y_{1},y_{2},...,y_{m})=f(x_{1},x_{2},...,x_{n}) \\tag{9}\n",
    "$$\n",
    "\n",
    "由于我们的已知基础函数$f$的求导规则，即f的雅可比矩阵是已知的。于是我们可以对$f$计算向量雅可比积(Vjp, Vector-jacobian-product)，并应用链式求导法则获得导数结果。\n",
    "\n",
    "$$\n",
    "\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial Y_{j}}{\\partial x_{1}}& ... & \\frac{\\partial Y_{j}}{\\partial x_{N}} \\\\\n",
    "  \\end{matrix}\n",
    "  \\right]=\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial Y_{j}}{\\partial y_{1}}& ... & \\frac{\\partial Y_{j}}{\\partial y_{m}} \\\\\n",
    "  \\end{matrix}\n",
    "  \\right]\\left[\n",
    " \\begin{matrix}\n",
    "   \\frac{\\partial y_{1}}{\\partial x_{1}}& ... & \\frac{\\partial y_{1}}{\\partial x_{n}} \\\\\n",
    "   ... & ... & ... \\\\\n",
    "   \\frac{\\partial y_{m}}{\\partial x_{1}} & ... & \\frac{\\partial y_{m}}{\\partial x_{n}}\n",
    "  \\end{matrix}\n",
    "  \\right]\n",
    "\\tag{10}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `grad`实现\n",
    "\n",
    "MindSpore中`grad`使用的是反向自动微分模式，即从正向网络的输出开始计算梯度。\n",
    "\n",
    "### `grad`算法设计\n",
    "\n",
    "设模型定义的原函数为：\n",
    "\n",
    "$$\n",
    "f(g(x, y, z)) \\tag{11}\n",
    "$$\n",
    "\n",
    "则$f()$对$x$的梯度为：\n",
    "\n",
    "$$\n",
    "\\frac{df}{dx}=\\frac{df}{dg}\\frac{dg}{dx}\\frac{dx}{dx}+\\frac{df}{dg}\\frac{dg}{dy}\\frac{dy}{dx}+\\frac{df}{dg}\\frac{dg}{dz}\\frac{dz}{dx}\\tag{12}\n",
    "$$\n",
    "\n",
    "$\\frac{df}{dy}$和$\\frac{df}{dz}$与$\\frac{df}{dx}$类似。\n",
    "\n",
    "应用链式求导法则，对每个函数(包括算子和图)定义梯度函数`bprop: dout->(df, dinputs)`，这里`df`表示函数对自由变量（函数外定义的变量）的梯度，`dinputs`是对函数输入的梯度。在此基础上，应用全微分法则，将`(df, dinputs)`累加到对应的变量。\n",
    "\n",
    "MindIR实现了分支，循环，闭包的函数表达式，所以对相应的算子实现正确的反向规则即可求得输入函数的梯度函数。\n",
    "\n",
    "定义运算符K，反向自动微分算法可以简单表示如下：\n",
    "\n",
    "```text\n",
    "v = (func, inputs)\n",
    "F(v): {\n",
    "    (result, bprop) = K(func)(K(inputs))\n",
    "    df, dinputs = bprop(dout)\n",
    "    v.df += df\n",
    "    v.dinputs += dinputs\n",
    "}\n",
    "```\n",
    "\n",
    "### `grad`算法实现\n",
    "\n",
    "在自动微分流程中，需要进行自动微分的函数会被取出。并作为自动微分模块的输入，并输出对应的梯度图。\n",
    "\n",
    "MindSpore的自动微分模块实现了从原函数对象到梯度函数对象的转换。转换后的对象为`fprop`形式的梯度函数对象。\n",
    "\n",
    "`fprop = (forward_result, bprop)`、`forward_result`是前向计算图的输出节点， `bprop`是以`fprop`的闭包对象形式生成的梯度函数，它只有`dout`一个入参， `inputs`和`outputs`是引用的`fprop`的输入和输出。\n",
    "\n",
    "```c++\n",
    "  MapObject();    // 实现ValueNode/Parameter/FuncGraph/Primitive对象的映射\n",
    "  MapMorphism();  // 实现CNode的态射\n",
    "  res = k_graph(); // res就是梯度函数的fprop对象\n",
    "```\n",
    "\n",
    "在生成梯度函数对象的过程中，需要完成从原函数到梯度函数的一系列的映射， 即为每个原函数中的节点生成其所对应的梯度函数的节点，再按照反向自动微分的规则将这些节点连接在一起，生成梯度函数图。\n",
    "\n",
    "每张原函数对象的子图都会都会生成一个`Dfunctor`对象，负责将该原函数对象映射为梯度函数对象。`DFunctor`主要需要经过 `MapObject`, `MapMorphism`两步来实现这种映射关系。\n",
    "\n",
    "`MapObject`实现了原函数节点到梯度函数节点的映射，具体包括对自由变量，参数节点以及ValueNode的映射。\n",
    "\n",
    "```c++\n",
    "MapFvObject();    // 自由变量的映射\n",
    "MapParamObject(); // 参数节点的映射\n",
    "MapValueObject(); // ValueNode的映射\n",
    "```\n",
    "\n",
    "- `MapFvObject`是对自由变量的映射；\n",
    "\n",
    "- `MapParamObject`是对参数节点的映射；\n",
    "\n",
    "- `MapValueObject`中主要对`Primitive`以及`FuncGraph`对象进行映射。\n",
    "\n",
    "其中，对`FuncGraph`进行的映射同样需要为该子图创造相应的`DFunctor`，是一个递归的过程。 `Primitive`表明了算子的种类，为了支持自动微分，需要为每一种`Primitive`定义其对应的反向微分函数。\n",
    "\n",
    "MindSpore将这些定义放在了Python侧，以`sin`算子为例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.ops as ops\n",
    "from mindspore.ops._grad.grad_base import bprop_getters\n",
    "\n",
    "@bprop_getters.register(ops.Sin)\n",
    "def get_bprop_sin(self):\n",
    "    \"\"\"Grad definition for `Sin` operation.\"\"\"\n",
    "    cos = ops.Cos()\n",
    "\n",
    "    def bprop(x, out, dout):\n",
    "        dx = dout * cos(x)\n",
    "        return (dx,)\n",
    "\n",
    "    return bprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`x`为原函数对象`sin`的输入，`out`为原函数对象`sin`的输出，`dout`为当前累加的梯度输入。\n",
    "\n",
    "当`MapObject`完成对以上节点的映射后，`MapMorphism`从原函数的输出节点开始以递归的方式实现对`CNode`的态射，建立起节点间的反向传播链接，实现梯度累加。\n",
    "\n",
    "### `grad`示例\n",
    "\n",
    "我们构建一个简单的网络来表示公式：\n",
    "\n",
    "$$\n",
    "f(x) = cos(sin(x)) \\tag{13}\n",
    "$$\n",
    "\n",
    "并对公式(13)的输入`x`进行求导：\n",
    "\n",
    "$$\n",
    "f'(x) = -sin(sin(x)) * cos(x) \\tag{14}\n",
    "$$\n",
    "\n",
    "在MindSpore中公式(13)的网络的结构实现为："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.nn as nn\n",
    "\n",
    "class Net(nn.Cell):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.sin = ops.Sin()\n",
    "        self.cos = ops.Cos()\n",
    "\n",
    "    def construct(self, x):\n",
    "        a = self.sin(x)\n",
    "        out = self.cos(a)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "正向网络的结构为：\n",
    "\n",
    "![auto-gradient-foward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/images/auto_gradient_foward.png)\n",
    "\n",
    "对该网络进行反向微分后，所得微分网络结构为：\n",
    "\n",
    "![auto-gradient-forward2](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/images/auto_gradient_forward2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 前向自动微分实现\n",
    "\n",
    "除了支持反向自动微分的`grad`之外，MindSpore还扩展实现了前向自动微分`jvp`（Jacobian-Vector-Product）。\n",
    "\n",
    "相比于反向自动微分，前向自动微分更适合于求取输入维度小于输出维度的网络的梯度。MindSpore的前向自动微分是基于反向自动微分接口`grad`开发的。\n",
    "\n",
    "![auto-gradient-jvp](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/r2.0.0-alpha/docs/mindspore/source_zh_cn/design/images/auto_gradient_jvp.png)\n",
    "\n",
    "黑色为网络的正向流程，第一次求导为针对$x$的求导，得到的是蓝色的图。第二次的为蓝色图针对$v$的求导，得到的是黄色的图。\n",
    "\n",
    "黄色的图就是我们所需要的前向模式自动微分的结果图。由于蓝色图可以视为关于$v$的线性函数，蓝色节点与黄色节点之间不会存在连边。蓝色节点全部为悬空节点，会被消除，真正运行的就只有原函数节点以及前向微分的节点。因此，该方法不会有额外的运行开销。\n",
    "\n",
    "### 参考文献\n",
    "\n",
    "[1] Baydin, A.G. et al., 2018. [Automatic differentiation in machine learning: A survey](https://arxiv.org/abs/1502.05767). arXiv.org. \\[Accessed September 1, 2021\\].\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore",
   "language": "python",
   "name": "mindspore"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

一键复制编辑原始数据按行查看历史

提交于 2023-02-02 10:13 . modify kernel name

https://gitee.com/mindspore/docs.git

git@gitee.com:mindspore/docs.git

mindspore

docs

r2.0.0-alpha

MindSpore / docs

简介

发行版

贡献者

近期动态

MindSpore / docs .gitee-modal { width: 500px !important; }

简介

发行版

开源评估指数源自 OSS-Compass 评估体系，评估体系围绕以下三个维度对项目展开评估：

贡献者

近期动态

搜索帮助

MindSpore / docs