# How to replicate the experiments

1. Either: *(i)* Use the files provided in this ZIP-file, including files from the GitHub repo *f-CMI* and *nnlib* by *hrayhar*, or,  
*(ii)* Download the repository f-CMI from https://github.com/hrayrhar/f-CMI and replace the files: *scripts/fcmi_scripts.py*, *scripts/fcmi_parse_results.py*, and *modules/bound_utils.py* with the corresponding files from the supplementary material. If errors occur in the next steps, it may also be necessary to replace *nnlib/nnlib/data_utils/abstract.py* with the corresponding file from the supplementary material. It may also be necessary to place the files *fcmi_scripts.py* and *fcmi_parse_results.py* directly in the *f-CMI* folder, rather than inside the folder *scripts*.

2. To replicate the experiments:
- To prepare the trained models, please run the `scripts/scripts.fcmi_train_classifier.py` script. For example:
    - MNIST: python -um scripts.fcmi_train_classifier -c configs/binary-mnist-4layer-CNN.json -d cuda -b 128 -e 200 -s 20 -v 10000 --exp_name fcmi-mnist-4vs9-CNN -D mnist --which_labels 4 9 --deterministic --n 75 --seed 0 --S_seed 0;
        - --seed: from 0 to 4, --S_seed: from 0 to 29. --n [75, 250, 1000, 4000].
    - CIFAR10: python -um scripts.fcmi_train_classifier -c configs/pretrained-resnet50-cifar10.json -d cuda -b 64 -e 40 -s 40 -v 10000 --exp_name cifar10-pretrained-resnet50 -D cifar10 -m StandardClassifier -A --resize_to_imagenet --optimizer sgd --lr 0.01 --momentum 0.9 --n 500 --seed 0 --S_seed 0; 
        - --seed: from 0 to 1, --S_seed: from 0 to 39. --n [500, 1000, 5000, 20000].
- Parse the result of the experiment using the "ecmi_parse_results.py" and "ecmi_parse_results_binning_binscale.py" scripts. The former is the experiments with the optimal number of bins, and the latter is that with various number of bins. For example:
    - (Non-recalibration): python ecmi_parse_results.py --exp_name fcmi-mnist-4vs9-CNN-LD --device cuda
    - (Recalibration with recalibration data): python ecmi_parse_results.py --exp_name fcmi-mnist-4vs9-CNN-LD --device cuda --recalibration True --recal_method 'hist_umb'
    - (Recalibration with training data; ours): python ecmi_parse_results.py --exp_name fcmi-mnist-4vs9-CNN-LD --device cuda --recalibration True --train_reuse True --recal_method 'hist_umb'


NOTE: We plan to make our source codes public at our GitHub.
