diff --git a/tutorials/source_en/advanced/compute_graph.md b/tutorials/source_en/advanced/compute_graph.md new file mode 100644 index 0000000000000000000000000000000000000000..06d16ab62418780aac127cc0587f752e63de2541 --- /dev/null +++ b/tutorials/source_en/advanced/compute_graph.md @@ -0,0 +1,177 @@ + + +# Computational Graph + +![comp-graph.png](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/advanced/images/comp-graph.png) + +A computational graph is a form of representing mathematical expressions by using directed graphs. As shown in the figure, the neural network structure can be regarded as a computational graph consisting of Tensor data and Tensor operations as nodes, so constructing a neural network and training it by using a deep learning framework is the process of constructing a computational graph and executing it. The current support for computational graphs in the industry framework is divided into two modes: dynamic graphs are executed by interpretation, with dynamic syntax affinity and flexible expression, and static graphs are executed by using JIT (just in time) compilation optimization. There are more restrictions on the syntax for using static syntax. + +MindSpore supports both computational graph modes with a unified API expression, using the same API in both modes and a unified automatic differentiation mechanism to achieve the unification of dynamic and static graphs. In the following, we introduce each of the two computational graph modes of MindSpore. + +## Dynamic Graphs + +Dynamic graphs are characterized by the fact that the construction and computation of a computational graph occur simultaneously (Define by run), which is consistent with Python interpreted execution. When a Tensor is defined in the computational graph, its value is calculated and determined, so it is easier to debug the model and get the value of intermediate results in real time. However, the need for all nodes to be saved makes it difficult to optimize the entire computational graph. + +In MindSpore, the dynamic graph pattern is also known as the PyNative pattern. Due to interpreted execution of dynamic graphs, it is recommended to use dynamic graph mode for debugging during script development and network process debugging. + +> Default computational graph mode in MindSpore is PyNative mode. + +If you need to manually control the framework to adopt PyNative mode, you can configure it with the following code: + +```python +import mindspore as ms + +ms.set_context(mode=ms.PYNATIVE_MODE) +``` + +In PyNative mode, the underlying operaors corresponding to all computation nodes is executed in a single Kernel, so that printing and debugging of computation results can be done arbitrarily, e.g. + +```python +import numpy as np +from mindspore import nn +from mindspore import ops +from mindspore import Tensor, Parameter + +class Network(nn.Cell): + def __init__(self): + super().__init__() + self.w = Parameter(Tensor(np.random.randn(5, 3), ms.float32), name='w') # weight + self.b = Parameter(Tensor(np.random.randn(3,), ms.float32), name='b') # bias + + def construct(self, x): + out = ops.matmul(x, self.w) + print('matmul: ', out) + out = out + self.b + print('add bias: ', out) + return out + +model = Network() +``` + +We simply define a Tensor with a shape of (5,) as input and observe the output. You can see that the `print` statement inserted in the `construct` method prints out the intermediate results in real time. + +```python +x = ops.ones(5, ms.float32) # input tensor +``` + +```python +out = model(x) +print('out: ', out) +``` + +```text +matmul: [-1.8809001 2.0400267 0.32370526] +add bias: [-1.6770952 1.5087128 0.15726662] +out: [-1.6770952 1.5087128 0.15726662] +``` + +## Static Graphs + +Compared to dynamic graphs, static graphs separate the construction of the computational graph from the actual computation (Define and run). In the build phase, the original computational graph is optimized and tuned according to the complete computational flow, and compiled to obtain a more memory-efficient and less computationally intensive computational graph. Since the structure of the graph does not change after compilation, it is called a "static graph". In the calculation phase, the results are obtained by executing the compiled calculation graph based on the input data. Compared with dynamic graphs, static graphs have a richer grasp of global information and can be optimized more, but their intermediate processes are black boxes for users, and they cannot get the intermediate calculation results in real time like dynamic graphs. + +In MindSpore, the static graph mode is also known as Graph mode. In Graph mode, based on graph optimization, computational graph whole graph sink and other techniques, the compiler can perform global optimization for the graph and obtain better performance, so it is more suitable for scenarios where the network is fixed and high performance is required. + +If you need to manually control the framework to adopt Graph mode, you can configure it with the following code: + +```python +ms.set_context(mode=ms.GRAPH_MODE) +``` + +### Graph Compilation Based on Source Code Conversion + +In static graph mode, MindSpore converts Python source code into Intermediate Representation IR (IR) by means of source code conversion, and based on this, optimizes the IR graph, and finally executes the optimized graph on hardware devices. MindSpore uses a functional IR based on graph representation, called MindIR. For details, see [Intermediate Representation MindIR](https://www.mindspore.cn/docs/en/master/design/mindir.html). + +MindSpore static graph execution process actually consists of two steps, corresponding to the Define and Run phases of the static graph. However, in practice, it is not sensed when the instantiated Cell object is called. MindSpore encapsulates both phases in the `__call__` method of the Cell, so the actual calling process is as follows: + +> `model(inputs) = model.compile(inputs) + model.construct(inputs)`, where `model` is the instantiated Cell object. + +We call the `compile` method explicitly for the following example: + +```python +model = Network() + +model.compile(x) +out = model(x) +print('out: ', out) +``` + +```text +out: [-0.26551223 3.0243678 0.706525 ] +``` + +### Static Graph Syntax + +In Graph mode, Python code is not executed by the Python interpreter, but the code is compiled into a static computational graph, and then the static computational graph is executed. Therefore, the compiler cannot support the full amount of Python syntax. MindSpore static graph compiler maintains a subset of Python common syntax to support the construction and training of neural networks. For details, refer to [Static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). + +### Static Graph Control Flow + +In PyNative mode, MindSpore fully supports flow control statements in Python native syntax. In Graph mode, MindSpore is compiled with performance optimizations, so there are some special constraints on the use of flow control statements when defining networks, but the rest remains consistent with the native Python syntax. For details, refer to [flow control statements](https://mindspore.cn/tutorials/experts/en/master/network/control_flow.html). + +## Just-in-time Compilation + +Usually, due to the flexibility of dynamic graphs, we choose to use the PyNative model for free neural network construction to achieve model innovation and optimization. But when performance acceleration is needed, we need to accelerate the neural network in part or as a whole. At this point, switching directly to Graph mode is an easy option, but the limitations of static graphs on syntax and control flow make it impossible to convert from dynamic to static graphs senselessly. + +For this purpose, MindSpore provides the `jit` decorator, which can make Python functions or Python-class member functions compiled into computational graphs, and improves the running speed by graph optimization and other techniques. At this point we can simply accelerate the graph compilation for the modules we want to optimize for performance, while the rest of the model, we still use the interpreted execution method, without losing the flexibility of dynamic graphs. + +### Cell Module Compilation + +When we need to speed up a part of the neural network, we can use the `jit` modifier directly on the `construct` method. The module is automatically compiled to a static graph when the instantiated object is called. The example is as follows: + +```python +import mindspore as ms +from mindspore import nn + +class Network(nn.Cell): + def __init__(self): + super().__init__() + self.fc = nn.Dense(10, 1) + + @ms.jit + def construct(self, x): + return self.fc(x) +``` + +### Function Compilation + +Similar to the Cell module compilation, when you need to compile acceleration for certain operations of Tensor, you can use `jit` modifier on its defined function. The module is automatically compiled as a static graph when the function is called. An example is as follows: + +> Based on the functional auto-differentiation feature of MindSpore, it is recommended to use the function compilation method for JIT compilation acceleration of Tensor operations. + +```python +@ms.jit +def mul(x, y): + return x * y +``` + +### Whole-graph Compilation + +MindSpore supports compiling and optimizing the forward computation, back propagation, and gradient optimization update of neural network training into one computational graph, which is called whole graph compilation. At this point, it only needs to construct the neural network training logic as a function and use the `jit` modifier on the function to achieve whole-graph compilation. + +The following is an example by using a simple fully connected network: + +```python +network = nn.Dense(10, 1) +loss_fn = nn.BCELoss() +optimizer = nn.Adam(network.trainable_params(), 0.01) + +def forward_fn(data, label): + logits = network(data) + loss = loss_fn(logits, label) + return loss + +grad_fn = ms.value_and_grad(forward_fn, None, optimizer.parameters) + +@ms.jit +def train_step(data, label): + loss, grads = grad_fn(data, label) + optimizer(grads) + return loss +``` + +As shown in the above code, after encapsulating the neural network forward execution and loss function as `forward_fn`, the function transformation is executed to obtain the gradient calculation function. Then the gradient calculation function and optimizer calls are encapsulated as `train_step` function and modified with `jit`. When the `train_step` function is called, the static graph is compiled, the whole graph is obtained and executed. + +In addition to using modifiers, `jit` methods can also be called by using function transformations, as in the following example: + +```python +train_step = ms.jit(train_step) +``` diff --git a/tutorials/source_en/advanced/compute_graph.rst b/tutorials/source_en/advanced/compute_graph.rst deleted file mode 100644 index f107eeb165c73139d1cebcfa90c6e45c4757e92b..0000000000000000000000000000000000000000 --- a/tutorials/source_en/advanced/compute_graph.rst +++ /dev/null @@ -1,8 +0,0 @@ -Computation Graphs -========================== - -.. toctree:: - :maxdepth: 1 - - compute_graph/mode - compute_graph/combine diff --git a/tutorials/source_en/advanced/compute_graph/combine.md b/tutorials/source_en/advanced/compute_graph/combine.md deleted file mode 100644 index 37559dbf7dd4b6cc3f411a0c324834019a804703..0000000000000000000000000000000000000000 --- a/tutorials/source_en/advanced/compute_graph/combine.md +++ /dev/null @@ -1,299 +0,0 @@ -# Combination of Dynamic and Static Graphs - - - -Currently, dynamic and static graphs are supported in the industry. Dynamic graphs are executed through explanation, with dynamic syntax affinity and flexible expression. Static graphs are executed through just in time (JIT) build, which focuses on static syntax and has many syntax constraints. The build process of the dynamic graph is different from that of the static graph. As a result, the syntax constraints are also different. - -For dynamic and static graph modes, MindSpore first unifies the API expression and uses the same APIs in the two modes. Then, it unifies the underlying differentiation mechanism of dynamic and static graphs. - -![dynamic](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/compute_graph/images/framework1.png) - -## Implementation Principle - -MindSpore allows you to use the `jit` modifier to modify objects that need to be executed using static graphs, achieving combination of dynamic and static graphs. The following uses a simple combination example to describe the implementation principle. The sample code is as follows: - -```python -import numpy as np -import mindspore.nn as nn -import mindspore as ms - -class Add(nn.Cell): - """Define a class to implement the x self addition.""" - def construct(self, x): - x = x + x - x = x + x - return x - -class Mul(nn.Cell): - """Define a class to implement the x self multiplication.""" - @ms.jit # Use jit to modify the function. This function is executed in static graph mode. - def construct(self, x): - x = x * x - x = x * x - return x - -class Test(nn.Cell): - """Define a class to implement Add(x), Mul(x), and then Add(x).""" - def __init__(self): - super(Test, self).__init__() - self.add = Add() - self.mul = Mul() - - def construct(self, x): - x = self.add(x) - x = self.mul(x) - x = self.add(x) - return x - -ms.set_context(mode=ms.PYNATIVE_MODE) -x = ms.Tensor(np.ones([3, 3], dtype=np.float32)) -print("init x:\n", x) -net = Test() -x = net(x) -print("\nx:\n", x) -``` - -```text - init x: - [[1. 1. 1.] - [1. 1. 1.] - [1. 1. 1.]] - - x: - [[1024. 1024. 1024.] - [1024. 1024. 1024.] - [1024. 1024. 1024.]] -``` - -According to the preceding information, after the test operation, the final value of x is a 3\*3 matrix whose each element is 8. The following figure shows the build method of this test case according to the execution sequence. - -![msfunction](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/compute_graph/images/jit.png) - -Functions modified by `jit` are built and executed in static graph mode. If the network involves reverse derivation, the part modified by `jit` is also used to generate a backward graph in the form of an entire graph. The backward graph is connected to backward graphs of operators before and after the graph and then delivered for execution. The cache policy is the same as that of the static graph. When the input shape and type information of the same function object is the same, the built graph structure is cached. - -## `jit` Modifier - -To improve the execution speed of forward computing tasks in dynamic graph mode, MindSpore provides the `jit` modifier. You can modify Python functions or member functions of Python classes to build them into computational graphs. Technologies such as graph optimization are used to improve the running speed. - -### Usage - -MindSpore supports static build in dynamic graphs. You can use the `jit` modifier to modify the function objects that need to be executed using static graphs to implement mixed execution of dynamic and static graphs. - -#### 1. Modifying Independent Function - -When using the `jit` modifier, you can modify an independently defined function so that it can run in static graph mode. The following is an example: - -```python -import numpy as np -import mindspore.ops as ops -import mindspore as ms - -# Set the running mode to dynamic graph mode. -ms.set_context(mode=ms.PYNATIVE_MODE) - -# Use the modifier to specify the execution in static graph mode. -@ms.jit -def add_func(x, y): - return ops.add(x, y) - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) - -out = add_func(x, y) -print(out) -``` - -```text - [5. 7. 9.] -``` - -In the preceding sample code, although the running mode is set to dynamic graph mode at the beginning, the `add_func(x, y)` function is modified using the `jit` modifier. Therefore, the `add_func(x, y)` function still runs in static graph mode. - -#### 2. Modifying the Member Functions of a Class - -When using the `jit` modifier, you can modify the member methods of classes. The sample code is as follows: - -```python -import numpy as np -import mindspore.nn as nn -import mindspore.ops as ops -import mindspore as ms - -# Set the running mode to dynamic graph mode. -ms.set_context(mode=ms.PYNATIVE_MODE) - -class Add(nn.Cell): - - @ms.jit # Use the modifier to specify the execution in static graph mode. - def construct(self, x, y): - out = x + y - return out - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) - -net = Add() -grad_ops = ms.grad(net, grad_position=(0, 1)) # Define the derivation operation. -grad_out = grad_ops(x, y) - -print("Infer result:\n", net(x, y)) - -print("Gradient result:") -print("Grad x Tensor1:\n", grad_out[0]) # Derivation of x -print("Grad y Tensor2:\n", grad_out[1]) # Derivation of y -``` - -```text - Infer result: - [5. 7. 9.] - Gradient result: - Grad x Tensor1: - [1. 1. 1.] - Grad y Tensor2: - [1. 1. 1.] -``` - -According to the preceding information, the sum of x and y is \[5, 7, 9\]. The derivation result of x is the same as that of y, that is, \[1, 1, 1\]. - -### Precautions - -When using `jit` to modify functions to improve execution efficiency, pay attention to the following points: - -1. Functions modified by `jit` must be within the syntax scope supported by static graph build, including but not limited to data types. - -2. The control flow syntax supported by a function modified by `jit` is the same as that supported by the static graph. An acceleration effect is achieved only for a control flow structure with a fixed loop count or a branch condition. - -3. When the `@jit` decorated function is used in PyNative mode, the parts that are not modified by `jit` support breakpoint debugging, and the parts modified by `jit` do not support breakpoint debugging because they are built in static graph mode. - -4. Functions modified by `jit` are built and executed in static graph mode. Therefore, `jit` decorator does not support the Hook operator in the modified functions or the customized Bprop function. - -5. Functions modified by `jit` are affected by side effects of static graph functions. Side effects of a function refer to the additional effects on the main function in addition to the return value of the function, for example, modifying global variables (variables other than the function) and modifying function parameters. - - Scenario 1: - - ```python - import numpy as np - import mindspore as ms - - # pylint: disable=W0612 - - value = 5 - - @ms.jit - def func(x, y): - out = x + y - value = 1 - return out - - ms.set_context(mode=ms.PYNATIVE_MODE) - x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) - y = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) - func(x, y) - print(value) - ``` - - ```text - 5 - ``` - - In this scenario, `value` is a global variable and is modified in the `func` function. In this case, if `jit` decorator is used to modify the `func` function, the global variable `value` is not changed. The reason is that statements irrelevant to return values are optimized during static graph build. - - Scenario 2: - - ```python - import numpy as np - import mindspore.nn as nn - import mindspore as ms - - class Func(nn.Cell): - def __init__(self): - super(Func, self).__init__() - self.value = 5 - - @ms.jit - def construct(self, x): - out = self.value + x - return out - - ms.set_context(mode=ms.PYNATIVE_MODE) - x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) - func = Func() - print("out1:", func(x)) - func.value = 1 - print("out2:", func(x)) - ``` - - ```text - out1: [6. 7. 8.] - out2: [6. 7. 8.] - ``` - - According to the preceding information, after the member variable `value` of the `func` class is changed to 1, the `construct` operation of the member function is not affected. In this scenario, `jit` is used to modify the `construct` member function of the `func` object. When `construct` is executed, it is built and executed in static graph mode. The static graph caches the build result. Therefore, when `func` is called for the second time, the modification of `value` does not take effect. - -6. If a function with the `jit` modifier contains operators (such as `MatMul` and `Add`) that do not require parameter training, these operators can be directly called in the modified function. If the modified function contains operators (such as `Conv2D` and `BatchNorm` operators) that require parameter training, these operators must be instantiated outside the modified function. The following uses sample code to describe the two scenarios. - - Scenario 1: Directly call an operator (`mindspore.ops.Add` in the example) that does not require parameter training in the modified function. The sample code is as follows: - - ```python - import numpy as np - import mindspore as ms - import mindspore.ops as ops - - ms.set_context(mode=ms.PYNATIVE_MODE) - - add = ops.Add() - - @ms.jit - def add_fn(x, y): - res = add(x, y) - return res - - x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) - y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) - z = add_fn(x, y) - - print("x:", x.asnumpy(), "\ny:", y.asnumpy(), "\nz:", z.asnumpy()) - ``` - - ```text - x: [1. 2. 3.] - y: [4. 5. 6.] - z: [5. 7. 9.] - ``` - - Scenario 2: The operator (`mindspore.nn.Conv2d` in the example) that requires parameter training must be instantiated outside the modified function. The sample code is as follows: - - ```python - import numpy as np - import mindspore.nn as nn - import mindspore as ms - - ms.set_context(mode=ms.PYNATIVE_MODE) - - # Instantiate the conv_obj operator in the conv_fn function. - conv_obj = nn.Conv2d(in_channels=3, out_channels=4, kernel_size=3, stride=2, padding=0) - conv_obj.init_parameters_data() - - @ms.jit - def conv_fn(x): - res = conv_obj(x) - return res - - input_data = np.random.randn(1, 3, 3, 3).astype(np.float32) - z = conv_fn(ms.Tensor(input_data)) - print(z.asnumpy()) - ``` - - ```text - [[[[ 0.00829158 -0.02994147] - [-0.09116832 -0.00181637]] - - [[-0.00519348 -0.02172063] - [-0.04015012 -0.02083161]] - - [[ 0.00608188 -0.01443425] - [-0.01468289 0.01200477]] - - [[ 0.00845292 0.00044869] - [-0.00361492 0.01993337]]]] - ``` diff --git a/tutorials/source_en/advanced/compute_graph/images/framework1.png b/tutorials/source_en/advanced/compute_graph/images/framework1.png deleted file mode 100644 index b49791d8eed61c164c8df9d087be7ae34e2c7f51..0000000000000000000000000000000000000000 Binary files a/tutorials/source_en/advanced/compute_graph/images/framework1.png and /dev/null differ diff --git a/tutorials/source_en/advanced/compute_graph/images/framework2.png b/tutorials/source_en/advanced/compute_graph/images/framework2.png deleted file mode 100644 index a961c2f661836bd7c96b99d2ac25c12488819948..0000000000000000000000000000000000000000 Binary files a/tutorials/source_en/advanced/compute_graph/images/framework2.png and /dev/null differ diff --git a/tutorials/source_en/advanced/compute_graph/images/jit.png b/tutorials/source_en/advanced/compute_graph/images/jit.png deleted file mode 100644 index af6c3bd342f5a8ba45b0b2ddda7bdf651efb75d3..0000000000000000000000000000000000000000 Binary files a/tutorials/source_en/advanced/compute_graph/images/jit.png and /dev/null differ diff --git a/tutorials/source_en/advanced/compute_graph/mode.md b/tutorials/source_en/advanced/compute_graph/mode.md deleted file mode 100644 index a1c71ad9b8445f8e1111183374e95d7d3a4e92b4..0000000000000000000000000000000000000000 --- a/tutorials/source_en/advanced/compute_graph/mode.md +++ /dev/null @@ -1,274 +0,0 @@ -# Dynamic and Static Graphs - - - -Currently, there are two execution modes of a mainstream deep learning framework: a static graph mode (Graph) and a dynamic graph mode (PyNative). - -- In static graph mode, when the program is built and executed, the graph structure of the neural network is generated first, and then the computation operations involved in the graph are performed. Therefore, in static graph mode, the compiler can achieve better execution performance by using technologies such as graph optimization, which facilitates large-scale deployment and cross-platform running. - -- In dynamic graph mode, the program is executed line by line according to the code writing sequence. In the forward execution process, the backward execution graph is dynamically generated according to the backward propagation principle. In this mode, the compiler delivers the operators in the neural network to the device one by one for computing, facilitating users to build and debug the neural network model. - -## Introduction to Dynamic and Static Graphs - -MindSpore provides a unified encoding mode for static and dynamic graphs, significantly enhancing compatibility between both types of graphs. This enables you to switch between the static and dynamic graph modes by changing only one line of code, eliminating the need to develop multiple sets of code. The dynamic graph mode is the default mode of MindSpore and is mainly used for debugging, and the static graph mode has more efficient execution performance and is mainly used for deployment. - -> When switching the running mode from dynamic graph to static graph, pay attention to the [static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). - -### Mode Selection - -You can configure the `context` parameter to control the program running mode. The differences between the dynamic graph mode and the static graph mode are as follows: - -- **Application scenario:** The network structure of a static graph needs to be built at the beginning, and then the framework optimizes and executes the entire graph. This mode is applicable to scenarios where the network is fixed and high performance is required. Operators are executed line by line on a dynamic graph. Single operator, common functions, and networks can be executed, and gradients can be computed separately. - -- **Network execution:** When the same network and operator are executed in static graph mode and dynamic graph mode, the accuracy effect is the same. The static graph mode uses technologies such as graph optimization and entire computational graph offloading. The static graph mode has higher network performance and efficiency, while the dynamic graph mode facilitates debugging and optimization. - -- **Code debugging:** The dynamic graph mode is recommended for script development and network process debugging. In dynamic graph mode, you can easily set breakpoints and obtain intermediate results of network execution. You can also debug the network in pdb mode. In static graph mode, breakpoints cannot be set. You can only specify an operator for printing and view the output result after the network execution is complete. - -### Mode Switching - -During mode switching, you need to set the running mode in the context. Define the network model `MyNet` and the data used in subsequent code snippets for subsequent switching and display of the dynamic and static graph modes. - -```python -import numpy as np -import mindspore.nn as nn -import mindspore.ops as ops -import mindspore as ms - -class MyNet(nn.Cell): - """Customize the network to implement the addition of two tensors.""" - def __init__(self): - super(MyNet, self).__init__() - self.add = ops.Add() - - def construct(self, x, y): - return self.add(x, y) - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) -``` - -Set the running mode to static graph mode. - -```python -import mindspore as ms - -ms.set_context(mode=ms.GRAPH_MODE) - -net = MyNet() -print(net(x, y)) -``` - -```text - [5. 7. 9.] -``` - -When MindSpore is in static graph mode, you can switch to the dynamic graph mode by setting `mode=ms.PYNATIVE_MODE`. Similarly, when MindSpore is in dynamic graph mode, you can switch to the static graph mode by setting`mode=ms.GRAPH_MODE`. Pay attention to [static graph syntax support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). - -```python -ms.set_context(mode=ms.PYNATIVE_MODE) - -net = MyNet() -print(net(x, y)) -``` - -```text - [5. 7. 9.] -``` - -## Static Graph - -In MindSpore, the static graph mode is also called the Graph mode, which is applicable to scenarios where the network is fixed and high performance is required. You can set the input parameter `mode` to `GRAPH_MODE` in the `set_context` API to set the static graph mode. - -In static graph mode, the compiler can perform global optimization on graphs based on technologies such as graph optimization and entire computational graph offloading. Therefore, good performance can be obtained when the compiler is executed in static graph mode. However, the execution graph is converted from the source code. Therefore, not all Python syntax is supported in static graph mode. There are some special constraints. For details about the support, see [Static Graph Syntax Support](https://www.mindspore.cn/docs/en/master/note/static_graph_syntax_support.html). - -### Execution Principle of the Static Graph Mode - -In static graph mode, MindSpore converts the Python source code into an intermediate representation (IR), optimizes the IR graph based on the IR, and executes the optimized graph on the hardware device. - -MindSpore uses a graph-based functional IR called MindIR. The static graph mode is built and optimized based on MindIR. When using the static graph mode, you need to use the [nn.Cell](https://mindspore.cn/docs/en/master/api_python/nn/mindspore.nn.Cell.html#mindspore.nn.Cell) class and writ execution code in the `construct` function. - -### Code Example in Static Graph Mode - -The code of the static graph mode is as follows. The neural network model implements the computation operation of $f(x, y)=x*y$. - -```python -# Set the running mode to static graph mode. -ms.set_context(mode=ms.GRAPH_MODE) - -class Net(nn.Cell): - """Customize the network to implement the multiplication of two tensors.""" - def __init__(self): - super(Net, self).__init__() - self.mul = ops.Mul() - - def construct(self, x, y): - """Define execution code."" - return self.mul(x, y) - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) - -net = Net() - -print(net(x, y)) -``` - -```text - [ 4. 10. 18.] -``` - -### Control Flow in Static Graph Mode - -For details about control flows in static graph mode, see [Process Control Statements](https://mindspore.cn/tutorials/experts/en/master/network/control_flow.html). - -## Dynamic Graph - -In MindSpore, the dynamic graph mode is also called the PyNative mode. You can set the input parameter `mode` to `PYNATIVE_MODE` in the `set_context` API to set the dynamic graph mode. - -During script development and network process debugging, you are advised to use the dynamic graph mode for debugging. The dynamic graph mode supports single-operator execution, common function execution, network execution, and independent gradient computation. - -### Execution Principle of the Dynamic Graph Mode - -In dynamic graph mode, you can use complete Python APIs. In addition, when APIs provided by MindSpore are used, the framework executes operator API operations on the corresponding hardware platform based on the selected hardware platform (Ascend/GPU/CPU) or environment information, and returns the corresponding result. - -The overall execution process of the framework is as follows: - -![process](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_en/advanced/compute_graph/images/framework2.png) - -The front-end Python API is called to the framework layer and finally computed on the corresponding hardware device. - -The following uses the `ops.mul` operator to replace the network model that needs to be defined in static graph mode to implement the computation of $f(x, y)=x*y$. - -```python -# Set the running mode to dynamic graph mode. -ms.set_context(mode=ms.PYNATIVE_MODE) - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -y = ms.Tensor(np.array([4.0, 5.0, 6.0]).astype(np.float32)) - -output = ops.mul(x, y) - -print(output.asnumpy()) -``` - -```text - [ 4. 10. 18.] -``` - -In the preceding sample code, when the `ops.mul(x, y)` API is called, the Python API at the MindSpore expression layer is called to the C++ layer of the MindSpore framework using [Pybind11](https://pybind11.readthedocs.io/en/stable/basics.html) and converted into the C++ API. Then, the framework automatically selects the corresponding hardware device based on the MindSpore installation environment information and performs the add operation on the hardware device. - -According to the preceding principles, in PyNative mode, Python script code is executed based on the Python syntax. During the execution, Python APIs at the MindSpore expression layer are executed on different hardware based on user settings to accelerate performance. - -Therefore, in dynamic graph mode, you can use Python syntax and debugging methods as required. - -### Principle of Automatic Differentiation in Dynamic Graph Mode - -In a dynamic graph, the forward propagation process is executed based on the Python syntax, and the backward propagation process is implemented based on tensors. - -Therefore, during the forward propagation process, all operations applied to tensors are recorded and computed backward, and all backward propagation processes are connected to form an overall backward propagation graph. Finally, the backward graph is executed on the device and the gradient is computed. - -The following uses a simple sample code to describe the principle of automatic differentiation in dynamic graph mode. Multiply the matrix x by a fixed parameter z, and then perform matrix multiplication with y: - -$$f(x, y)=(x * z) * y \tag{1}$$ - -The sample code is as follows: - -```python -# Set the running mode to dynamic graph mode. -ms.set_context(mode=ms.PYNATIVE_MODE) - -class Net(nn.Cell): - """Customize a network.""" - def __init__(self): - super(Net, self).__init__() - self.matmul = ops.MatMul() - self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z') - - def construct(self, x, y): - x = x * self.z - x = self.matmul(x, y) - return x - -class GradNetWrtX(nn.Cell): - """Define the derivation of x.""" - def __init__(self, net): - super(GradNetWrtX, self).__init__() - - self.net = net - - def construct(self, x, y): - gradient_function = ms.grad(self.net) - return gradient_function(x, y) - -x = ms.Tensor([[0.8, 0.6, 0.2], [1.8, 1.3, 1.1]], dtype=ms.float32) -y = ms.Tensor([[0.11, 3.3, 1.1], [1.1, 0.2, 1.4], [1.1, 2.2, 0.3]], dtype=ms.float32) - -output = GradNetWrtX(Net())(x, y) -print(output) -``` - -```text - [[4.5099998 2.7 3.6000001] - [4.5099998 2.7 3.6000001]] -``` - -> The accuracy may vary depending on the computing platform. Therefore, the execution results of the preceding code vary slightly on different platforms. For details about the derivation of the formula and the explanation of the preceding printed results, see [Automatic Derivation](https://www.mindspore.cn/tutorials/en/master/advanced/derivation.html#). - -![forward](https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/website-images/master/tutorials/source_zh_cn/advanced/compute_graph/images/forward_backward.png) - -It can be learned from the preceding dynamic graph mode that, in a forward propagation process, MindSpore records a computation process of Mul, and a backward MulGrad operator is obtained according to a definition of a backward bprop corresponding to Mul. - -The bprop definition of the Mul operator is as follows: - -```python -from mindspore.ops._grad.grad_base import bprop_getters - -@bprop_getters.register(ops.Mul) -def get_bprop_mul(self): - """Grad definition for `Mul` operation.""" - mul_func = P.Mul() - - def bprop(x, y, out, dout): - bc_dx = mul_func(y, dout) - bc_dy = mul_func(x, dout) - return binop_grad_common(x, y, bc_dx, bc_dy) - - return bprop -``` - -You can see that two backward propagation gradient values of the input and output are required to compute the backward input of Mul. In this case, you can connect z to MulGrad based on the actual input value. The rest can be deduced by analogy. For the next operator Matmul, the MatmulGrad information is obtained accordingly, and then the context gradient propagation is connected based on the input and output of bprop. - -Similarly, for derivation of input y, the same process may be used for derivation. - -### Control Flow in Dynamic Graph Mode - -In MindSpore, the control flow syntax is not specially processed. Instead, the control flow syntax is directly executed based on the Python syntax, and automatic differentiation operations are performed on the expanded execution operators. - -For example, in a for loop, the Python source code is executed first in the dynamic graph, and then the statements in the for loop are continuously executed based on the number of loops, and automatic differentiation operations are performed on the operators. - -```python -# Set the running mode to dynamic graph mode. -ms.set_context(mode=ms.PYNATIVE_MODE) - -class Net(nn.Cell): - """Customize a network.""" - def __init__(self): - super(Net, self).__init__() - self.matmul = ops.MatMul() - self.z = ms.Parameter(ms.Tensor(np.array([1.0], np.float32)), name='z') - - def construct(self, x): - for _ in range(3): - x = x + self.z - return x - -x = ms.Tensor(np.array([1.0, 2.0, 3.0]).astype(np.float32)) -net = Net() -output = net(x) - -print(output) -``` - -```text - [4. 5. 6.] -``` diff --git a/tutorials/source_zh_cn/advanced/compute_graph.ipynb b/tutorials/source_zh_cn/advanced/compute_graph.ipynb index 3a7ba7ec6e0b034d368bd3557ac556ad558789f2..057cbc17307c34eb01b831ddf5a437c23c804a48 100644 --- a/tutorials/source_zh_cn/advanced/compute_graph.ipynb +++ b/tutorials/source_zh_cn/advanced/compute_graph.ipynb @@ -138,7 +138,7 @@ "id": "4cd72031-836d-475e-a74d-5264b29393d1", "metadata": {}, "source": [ - "相较于动态图而言,静态图的特点是将计算图的构建和实际计算分开(Define and run)。在构建阶段,根据完整的计算流程对原始的计算图进行优化和调整,编译得到更省内存和计算量更少的计算图。由于编译之后图的结构不再改变,所以称之为 “静态图” 。 在计算阶段,根据输入数据执行编译好的计算图得到计算结果。相较于动态图,静态图对全局的信息掌握更丰富,可做的优化也会更多,但是其中间过程对于用户来说是黑盒,无法像动态图一样实时拿到中间计算结果。\n", + "相较于动态图而言,静态图的特点是将计算图的构建和实际计算分开(Define and run)。在构建阶段,根据完整的计算流程对原始的计算图进行优化和调整,编译得到更省内存和计算量更少的计算图。由于编译之后图的结构不再改变,所以称之为 “静态图” 。在计算阶段,根据输入数据执行编译好的计算图得到计算结果。相较于动态图,静态图对全局的信息掌握更丰富,可做的优化也会更多,但是其中间过程对于用户来说是黑盒,无法像动态图一样实时拿到中间计算结果。\n", "\n", "在MindSpore中,静态图模式又被称为Graph模式,在Graph模式下,基于图优化、计算图整图下沉等技术,编译器可以针对图进行全局的优化,获得较好的性能,因此比较适合网络固定且需要高性能的场景。\n", "\n",