Ascend
GPU
CPU
Data Preparation
When resource conditions permit, in order to pursue higher performance, data transformations are generally executed in the data pipeline mode. That is, users have to define the map
operator which helps to execute augmentations in data pipeline. As shown in the figure below, the map
operator contains 3 transformations: Resize
, Crop
, and HWC2CHW
. When the pipeline starts, the map
operator will apply these transformations to data in sequence.
Although the data pipeline can process input data quickly, the code of defining pipeline seems heavy while sometimes users just want to focus on the data transformations and perform them on small-scale data. In this case, data pipeline is not necessary.
Therefore, MindSpore provides a lightweight data processing way to execute these data augmentations, called Eager mode
.
In Eager mode
, the execution of data augmentations will not rely on the map
operator but can be called directly as callable functions. The code will be simpler since the results are obtained immediately. It is recommended to be used in lightweight scenarios such as small data enhancement experiments and model inference.
MindSpore currently supports executing various data augmentations in Eager mode
, as shown below. For more details, please refer to the API documentation.
Note: In chapters Image Processing and Enhancement, Text Processing and Enhancement, all data enhancement operators can be executed in Eager mode.
The following example introduces how to execute data augmentations in Eager mode
.
To use
Eager mode
, just treat the data augmentations as an executable function and call them directly.
Download the image and save it to the specified location.
import os
import requests
requests.packages.urllib3.disable_warnings()
def download_dataset(dataset_url, path):
filename = dataset_url.split("/")[-1]
save_path = os.path.join(path, filename)
if os.path.exists(save_path):
return
if not os.path.exists(path):
os.makedirs(path)
res = requests.get(dataset_url, stream=True, verify=False)
with open(save_path, "wb") as f:
for chunk in res.iter_content(chunk_size=512):
if chunk:
f.write(chunk)
print("The {} file is downloaded and saved in the path {} after processing".format(os.path.basename(dataset_url), path))
download_dataset("https://mindspore-website.obs.cn-north-4.myhuaweicloud.com/notebook/datasets/banana.jpg", ".")
This example will mix the c_tranforms
and py_transforms
operators from the vision
module to transform a given image.
You only need to focus on what data augmentations have to use, not any code for the data pipeline.
The Eager mode of the vision operator supports numpy.array
or PIL.Image
type data as input parameters.
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
import mindspore.dataset.vision.c_transforms as C
import mindspore.dataset.vision.py_transforms as P
img_ori = Image.open("banana.jpg").convert("RGB")
print("Image.type: {}, Image.shape: {}".format(type(img_ori), img_ori.size))
# Define a Resize op from c_transform and execute it immediately
op1 = C.Resize(size=(320))
img = op1(img_ori)
print("Image.type: {}, Image.shape: {}".format(type(img), img.shape))
# Define a CenterCrop op from c_transform and execute it immediately
op2 = C.CenterCrop((280, 280))
img = op2(img)
print("Image.type: {}, Image.shape: {}".format(type(img), img.shape))
# Define a Pad op from py_transform and execute it immediately
# Before calling Pad, you need to call ToPIL()
op3 = P.ToPIL()
op4 = P.Pad(40)
img = op4(op3(img))
print("Image.type: {}, Image.shape: {}".format(type(img), img.size))
# Show the result
plt.subplot(1, 2, 1)
plt.imshow(img_ori)
plt.title("original image")
plt.subplot(1, 2, 2)
plt.imshow(img)
plt.title("transformed image")
plt.show()
The output is as follows:
Image.type: <class 'PIL.Image.Image'>, Image.shape: (356, 200)
Image.type: <class 'numpy.ndarray'>, Image.shape: (320, 570, 3)
Image.type: <class 'numpy.ndarray'>, Image.shape: (280, 280, 3)
Image.type: <class 'PIL.Image.Image'>, Image.shape: (360, 360)
The following shows the processed image.
Augmentation operators that support to be run in Eager Mode are listed as follows: mindspore.dataset.transforms, mindspore.dataset.vision, mindspore.dataset.text.transforms.
This example will transform the given text using the tranforms
operator in the text
module.
Eager mode of the text operator supports numpy.array
type data as input parameters.
import mindspore.dataset.text.transforms as text
from mindspore import dtype as mstype
# Define a WhitespaceTokenizer op and execute it immediately
txt = "Welcome to Beijing !"
txt = text.WhitespaceTokenizer()(txt)
print("Tokenize result: {}".format(txt))
# Define a ToNumber op and execute it immediately
txt = ["123456"]
to_number = text.ToNumber(mstype.int32)
txt = to_number(txt)
print("ToNumber result: {}, type: {}".format(txt, type(txt[0])))
Tokenize result: ['Welcome' 'to' 'Beijing' '!']
ToNumber result: [123456], type: <class 'numpy.int32'>
This example will transform the given data using the operators of c_tranforms
in the transforms
module.
Eager mode of transforms operator supports numpy.array
type data as input parameters.
import numpy as np
import mindspore.dataset.transforms.c_transforms as trans
# Define a Fill op and execute it immediately
data = np.array([1, 2, 3, 4, 5])
fill = trans.Fill(0)
data = fill(data)
print("Fill result: ", data)
# Define a OneHot op and execute it immediately
label = np.array(2)
onehot = trans.OneHot(num_classes=5)
label = onehot(label)
print("OneHot result: ", label)
Fill result: [0 0 0 0 0]
OneHot result: [0 0 1 0 0]
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。