# msit_3

**Repository Path**: eveyin/msit_3

## Basic Information

- **Project Name**: msit_3
- **Description**: 统一推理工具链入口，提供客户一体化开发工具，支持一站式调试调优
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 465
- **Created**: 2024-08-07
- **Last Updated**: 2024-12-07

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

【review】1: 第11行，SparseMethods 使用了 namedtuple 来定义，但后续代码中并没有充分利用 namedtuple 的特性。建议直接使用字符串常量来定义 SparseMethods，以简化代码并提高可读性。

python
复制
SparseMethods = ('depth_growth', 'width_growth')
【review】2: 第20行，security.check_type 和 security.check_int 等方法的参数检查可以合并到一个方法中，以减少重复代码。建议创建一个通用的参数检查方法。

python
复制
def check_param(value, expected_type, param_name, min_value=None):
    if not isinstance(value, expected_type):
        raise TypeError(f"{param_name} should be of type {expected_type}, but got {type(value)}")
    if min_value is not None and value < min_value:
        raise ValueError(f"{param_name} should be at least {min_value}, but got {value}")
【review】3: 第35行，count_parameters 方法中的列表推导式可以简化，以提高代码的可读性。

python
复制
def count_parameters(self):
    return sum(np.prod(list(param.shape)) for group in self._optimizer.param_groups for param in group['params'])
【review】4: 第50行，reset 方法中对 param_groups 的深拷贝可以优化，避免不必要的深拷贝操作。

python
复制
self._optimizer.param_groups = []
param_groups = deepcopy(self._param_groups_args)
可以改为：

python
复制
self._optimizer.param_groups = deepcopy(self._param_groups_args)
【review】5: 第80行，SparseForward 类的初始化方法中，对 epochs_each_stage 的检查可以合并到一个方法中，以减少重复代码。

python
复制
def check_epochs_each_stage(epochs_each_stage):
    if not isinstance(epochs_each_stage, (list, tuple)):
        raise TypeError("epochs_each_stage should be a list or tuple")
    if len(epochs_each_stage) < 2:
        raise ValueError("epochs_each_stage should have at least two elements")
    for epoch in epochs_each_stage[:-1]:
        if not isinstance(epoch, int) or epoch <= 0:
            raise ValueError("Each element in epochs_each_stage (except the last one) should be a positive integer")
    if epochs_each_stage[-1] != -1 and (not isinstance(epochs_each_stage[-1], int) or epochs_each_stage[-1] <= 0):
        raise ValueError("The last element in epochs_each_stage should be -1 or a positive integer")
【review】6: 第110行，scale_model 方法中，dummy_inputs 的创建可以简化，避免重复代码。

python
复制
dummy_inputs = [torch.ones([1] + list(input_shape)[1:]) for input_shape in input_shapes]
if self._is_on_npu:
    dummy_inputs = [dummy_input.npu() for dummy_input in dummy_inputs]
可以改为：

python
复制
dummy_inputs = [torch.ones([1] + list(input_shape)[1:]).to(model.device) for input_shape in input_shapes]
【review】7: 第130行，get_cur_stage 方法中，self._global_batch_id 的更新可以放在方法外部，以避免每次调用时都更新。

python
复制
def get_cur_stage(self):
    cur_stage = np.sum(self._cum_steps_each_stage < self._global_batch_id)
    logger.debug(f"SparseForward global_batch_id: {self._global_batch_id}")
    return cur_stage

def update_global_batch_id(self):
    self._global_batch_id += 1
【review】8: 第150行，sparse_model_width 和 sparse_model_depth 函数中，SparseForward 的初始化可以提取到一个公共函数中，以减少重复代码。

python
复制
def init_sparse_forward(model, optimizer, steps_per_epoch, epochs_each_stage, sparse_mode):
    model.forward = SparseForward(model, optimizer, steps_per_epoch, epochs_each_stage, sparse_mode=sparse_mode)
    return model

def sparse_model_width(model, optimizer, steps_per_epoch, epochs_each_stage):
    return init_sparse_forward(model, optimizer, steps_per_epoch, epochs_each_stage, SparseMethods.WIDTH_GROWTH)

def sparse_model_depth(model, optimizer, steps_per_epoch, epochs_each_stage):
    return init_sparse_forward(model, optimizer, steps_per_epoch, epochs_each_stage, SparseMethods.DEPTH_GROWTH)
【review】9: 第170行，sparse_model_width 和 sparse_model_depth 函数的文档字符串中，示例代码可以放在 doctest 模块中进行测试，以确保示例代码的正确性。

python
复制
def sparse_model_width(model, optimizer, steps_per_epoch, epochs_each_stage):
    """ Wrapper a model for width sparse training
    Args:
      model: initialized torch model. `torch.nn.Module` instance.
      optimizer: initialized PyTorch optimizer.
      steps_per_epoch: int value for steps in each training repoch.
        The last element can be `-1` means training till total epochs end.
      epochs_each_stage: list or tuple value for epochs in each stage.
        For value like `[10, 20, -1]` means 3 training stages:
        - 1st stage, prune model width to `1/4` as the start model and train 10 epochs.
        - 2nd stage, double start model in the 1st stage to `1/2` as original model and train 20 epochs.
        - 3rd stage, double model in the 2nd stage to same as the original model.
          Epoch number `-1` means training till total epochs end.

    Return
      - model with `forward` being replaced by `SparseForward`.

    Examples:
    >>> import torch
    >>> from torchvision import models
    >>> from msmodelslim.pytorch.sparse import sparse_model_width
    >>> model = models.vgg16()
    >>> optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4)
    >>> model = sparse_model_width(model, optimizer, steps_per_epoch=100, epochs_each_stage=[1, 2, 1])
    >>> _ = model.train()
    >>> print("output shape:", model(torch.ones([1, 3, 224, 224])).shape)
    >>> # msmodelslim-logger - INFO - SparseForward Stage: -1 -> 0, scale: 4
    >>> # msmodelslim-logger - INFO - Previous model parameters: 138357544
    >>> # msmodelslim-logger - INFO - Previous optimizer parameters: 138357544
    >>> # msmodelslim-logger - INFO - New model parameters: 9418936
    >>> # msmodelslim-logger - INFO - New optimizer parameters: 9418936
    >>> # output shape: torch.Size([1, 1000])
    """
    return init_sparse_forward(model, optimizer, steps_per_epoch, epochs_each_stage, SparseMethods.WIDTH_GROWTH)
【review】10: 第190行，sparse_model_depth 函数的文档字符串中，示例代码可以放在 doctest 模块中进行测试，以确保示例代码的正确性。

python
复制
def sparse_model_depth(model, optimizer, steps_per_epoch, epochs_each_stage):
    """ Wrapper a model for depth sparse training
    Args:
      model: initialized torch model. `torch.nn.Module` instance.
      optimizer: initialized PyTorch optimizer.
      steps_per_epoch: int value for steps in each training repoch.
        The last element can be `-1` means training till total epochs end.
      epochs_each_stage: list or tuple value for epochs in each stage.
        For value like `[10, 20, -1]` means 3 training stages:
        - 1st stage, prune model depth to `1/4` as the start model and train 10 epochs.
        - 2nd stage, double start model in the 1st stage to `1/2` as original model and train 20 epochs.
        - 3rd stage, double model in the 2nd stage to same as the original model.
          Epoch number `-1` means training till total epochs end.

    Return model with `forward` being replaced by `SparseForward`

    Examples:
    >>> import torch
    >>> from torchvision import models
    >>> from msmodelslim.pytorch.sparse import sparse_model_depth
    >>> model = models.vit_b_16()
    >>> opt = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9, weight_decay=1e-4)
    >>> model = sparse_model_depth(model, opt, steps_per_epoch=100, epochs_each_stage=[1, 2, 1])
    >>> _ = model.train()
    >>> print("output shape:", model(torch.ones([1, 3, 224, 224])).shape)
    >>> # msmodelslim-logger - INFO - SparseForward Stage: -1 -> 0, scale: 4
    >>> # msmodelslim-logger - INFO - Previous model parameters: 86567656
    >>> # msmodelslim-logger - INFO - Previous optimizer parameters: 86567656
    >>> # msmodelslim-logger - INFO - New model parameters: 22776808
    >>> # msmodelslim-logger - INFO - New optimizer parameters: 22776808
    >>> # output shape: torch.Size([1, 1000])
    """
    return init_sparse_forward(model, optimizer, steps_per_epoch, epochs_each_stage, SparseMethods.DEPTH_GROWTH)