# KED

**Repository Path**: sari-smartcity/ked

## Basic Information

- **Project Name**: KED
- **Description**: KED-project
- **Primary Language**: Python
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-01-10
- **Last Updated**: 2026-01-10

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# pytorch-be-your-own-teacher
An pytorch implementation of paper 'Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation', https://arxiv.org/abs/1905.08094

## Introduction
We provide code of training ResNet18 and ResNet50 with multiple breanches on CIFAR100 dataset. 

## Dependencies:

+ Ubuntu 18.04.5 LTS
+ Python 3.6.9
+ PyTorch 1.7.1
+ torchvision 0.8.2 
+ numpy 1.19.2 
+ tensorboardX 2.1

Note: this is my machine environment, and the other version of software may also works.

## Train an ResNet on CIFAR-100:

```sh
# resnet18
python train.py train multi_resnet18_kd --data-dir /PATH/TO/CIFAR100 
python ex4.py --cmd train --arch multi_resnet18_kd --data-dir /PATH/TO/CIFAR100 --flag 515_path6 --dataset cifar100
python ex5_2.py --cmd train --arch multi_resnet18_kd  --data-dir /PATH/TO/CIFAR10 --flag 017_cifar10_18 --dataset cifar10
python sadex.py --cmd train --arch multi_resnet50_kd --data-dir /PATH/TO/CIFAR100 --flag sad50 --dataset cifar100
python ex6.py --cmd train --arch multi_resnet18_kd --data-dir /PATH/TO/CIFAR100 --flag 0708_at --dataset cifar100
python ked.py --cmd train --arch multi_resnet18_kd  --data-dir /PATH/TO/CIFAR10 --flag 017_cifar10_18 --dataset cifar10
python ked.py --cmd train --arch multi_resnet18_kd  --data-dir /PATH/TO/CIFAR100 --flag 017_cifar100_18 --dataset cifar100
#resnet34
python ked.py --cmd train --arch multi_resnet34_kd  --data-dir /PATH/TO/CIFAR10 --flag 017_cifar10_34 --dataset cifar10
python ked.py --cmd train --arch multi_resnet34_kd  --data-dir /PATH/TO/CIFAR100 --flag 017_cifar100_34 --dataset cifar100
# resnet50
python ex5_2.py --cmd train --arch multi_resnet50_kd  --data-dir /PATH/TO/CIFAR10 --flag 515_cifar10_50 --dataset cifar10
python ked.py --cmd train --arch multi_resnet50_kd  --data-dir /PATH/TO/CIFAR10 --flag 017_cifar10_50 --dataset cifar10
python ked.py --cmd train --arch multi_resnet50_kd  --data-dir /PATH/TO/CIFAR100 --flag 017_cifar10_50 --dataset cifar100
```

## Load an ResNet on CIFAR-100 and test it:
```sh
# resnet18
python train.py --cmd test --arch multi_resnet18_kd --data-dir /PATH/TO/CIFAR100 --resume ./save_checkpoints/multi_resnet18_kd_ex/model_best.path.tar

# resnet50
python train.py test multi_resnet50_kd --data-dir /PATH/TO/CIFAR100 --resume /save_checkpoints/multi_resnet18_kd/model_best.pth.tar
python train.py --cmd test --arch multi_resnet50_kd --data-dir /PATH/TO/CIFAR100 --resume ./save_checkpoints/multi_resnet50_kd_ex/model_best.path.tar
```

## Result on Resnet18

As the original paper does not tell us the hyper-parameters, I just use the follow setting. If you find better hyper-parmeters, you could tell me in the issues. Moreover, we do not know the side-branch architecture details. So the accuracy is lower than the original paper.
I will fine-tune the hyper-parameters.

21.01.12) At the same settings, with different library version, the performance of last classifier is better than paper result.
However, hyperparameter searching is still needed.

 ```
alpha = 0.1
temperature = 3
beta = 1e-6
```
|   Method   | Classifier 1/4 | Classifier 2/4 | Classifier 3/4 | Classifier 4/4 |
|:----------:|:--------------:|:--------------:|:--------------:|:--------------:|
| Original   |    **67.85**       |74.57          |       **78.23**   |     78.64      |


## Result on Resnet50

```
alpha = 0.1
temperature = 3
beta = 1e-6
```
|   Method   | Classifier 1/4 | Classifier 2/4 | Classifier 3/4 | Classifier 4/4 |
|:----------:|:--------------:|:--------------:|:--------------:|:--------------:|
| Original   |    68.23       |74.21          |       75.23   |     80.56      |