# Code Release 

Here is a code release for the 2023 NeurIPS submission "How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model". This code will be released via a GitHub repo (in a more polished form) if accepted. 

The code release is structured as follows:
- `circuit_discovery.py` reproduces the circuit discovery and semantics assignment process
- `big_ds_experiments.py` reproduces the experiments run on the larger, 10000 element dataset
- `neuron_investigations.py` reproduces the neuron-level experiments
- `sequence_generalization.py` reproduces the generalization experiments

The aforementioned files will cache files (in `paper-cache`) that can be used to generate plots using these scripts: 
- `circuit_discovery_plotting.py`
- `pca_plots.py`
- `neuron_plots.py`
- `appendix_plots.py`

In addition, we include three useful files in the `cache` folder (indices of the relevant logits, the nouns used in our template, and the order of MLP10 neurons, which otherwise takes a long time to compute). We also include two utility files, `utils.py` and `color_utils.py` (for plotting), as well as one file for working with the dataset object (`dataset.py`).

## Running the code
Unfortunately, using the `rust-circuit` library to work with `gpt2-small` is not easy. It will probably be more instructive to read the code directly than to actually run it. However, if you wish to do so, follow these steps:

1. Compile [rust-circuit](https://github.com/redwoodresearch/rust_circuit_public), following the instructions there given; note that this requires clang and rust. The repo instructs you to install maturin; be sure to install 0.14.x (we used 0.14.7), as newer versions do not work.
2. Install this project's requirements via the provided requirements file `pip install -r requirements.txt`
3. Download the `gpt2-small` model files from [this link](https://rrserve.s3.us-west-2.amazonaws.com/remix/remix_tensors.zip). Extract them to a folder called `../rrfs/tensor_db`