# KJ600

**Repository Path**: lnwbzmt/kj600

## Basic Information

- **Project Name**: KJ600
- **Description**: a LLM pretrain early warning system with built-in model's torch module and optimizer status collection and aggregation. 
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 4
- **Created**: 2024-03-30
- **Last Updated**: 2024-03-30

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

## Ascend/ModelLink 使用方法:

1. llama2
In ModelLink/pretrain_gpt.py, 
def model_provider(pre_process=True, post_process=True) -> Union[GPTModel, megatron.model.GPTModel]:
after the initialization of GPTModel, add following code.

config = {
    "targets": {
    "language_model.encoder.layers.0": {"input": "tuple[2]:0", "output": "tensor::"},         
    }
}
    
from module_hook import ModuleHook
ModuleHook().hook_modules(list(config["targets"].keys()), model=model, 
                              global_batch_size=args.global_batch_size, 
                              dp=args.data_parallel_size, 
                              micro_batch_size=args.micro_batch_size, fwd_or_bkd=0)