##### Transformer-base:

dense

```
cd 'Transformer-base/dense'
sh scripts/run_3090.sh
pip install -e . && sh scripts/run_3090.sh
```

S-STE

```
cd 'Transformer-base/STE+soft-thresholding+MVUE24(grad)+fp8'
pip install -e . && sh scripts/run_3090.sh
```

##### GPT-2:

dense

```
cd 'nanoGPT/gpt2-124M-fp16'
torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --compile=False
```

S-STE

```
cd 'gpt2-124M-STE+soft-thresholding+MVUE24(grad)+fp8'
torchrun --standalone --nproc_per_node=8 train.py config/train_gpt2.py --compile=False
```

##### DeiT-small:

dense

```
cd 'Deit-small/origin'
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model deit_small_patch16_224 --batch-size 256 --data-path /path/to/imagenet --output_dir /path/to/save
```

S-STE

```
cd 'Deit-small/STE+soft-thresholding+MVUE24(grad)+fp8'
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model deit_small_patch16_224 --batch-size 256 --data-path /path/to/imagenet --output_dir /path/to/save
```

