# Metric Space Magnitude for Evaluating the Diversity of Latent Representations

This repository is the official implementation of "Metric Space Magnitude for Evaluating the Diversity of Latent Representations".

## Structure

This repository contains an implementation of magnitude computations and the novel magnitude-based diversity measures detailed in our paper.

magdiv\distances.py - computes distance matrice.

magdiv\magnitude.py - computes the magnitude of a metric space.

magdiv\function_utils.py - contains helper functions for dealing with magnitude functions.

magdiv\summaries.py - computes our proposed diversity measures MagArea and MagDiff.

magdiv\magnipy.py - contains a class implementing all of the above magnitude computations.

magdiv\diversity_metrics.py - implements other diversity metrics as well as the computation of all diversity measures from embeddings.

The remaining scrips implement the experiments.

---


## Requirements

To install requirements we use the Python package manager poetry. Simply run:

```setup
poetry install
```

## Experiments

Data should be collected and saved as follows. We further explain how to reproduce each experiment.

## Overview Figures

The following Jupyter notebook reproduces the overview plots and toy examples from our paper.

```
./experiment0_toy_examples.ipynb
```

## Curvature Prediction

Download the pickle files point_clouds.pkl and labels.pkl from https://github.com/renata-turkes/turkevs2022on/tree/main/DATASETS/curvature into 
```
./data/curvature/
```

Then run the experiment using 
```
poetry run python3 experiment1_curvature.py
```

The prediction results for each method for predicting curvature will be printed.

**Reference:**
Turkes, Renata, Guido F. Montufar, and Nina Otter. "On the effectiveness of persistent homology." Advances in Neural Information Processing Systems 35 (2022): 35432-35448.

## Intrinsic Text Diversity Evaluation

Download and save the data by Trevet et. al. (2021) 
https://github.com/GuyTevet/diversity-eval
via the following link 
http://diversity-eval.s3-us-west-2.amazonaws.com/data.zip.

Unzip the files into 

```
../magdiv/data/dec/
```

Then run the experiment using 
```
poetry run python3 experiment2_dec_experiment.py
```

Results then can be visualised via the following notebook

```
experiment2_dec_results.ipynb
```

**Reference:**
Tevet, Guy, and Jonathan Berant. "Evaluating the Evaluation of Diversity in Natural Language Generation." Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.

## Characterising Document Embeddings

Follow the instructions detailed in the extended experiments and described by Wayland et. al. (2024) to download and embed the first 2^14 samples of four huggingface datasets using six different embedding models. Save each embedding as
```
model_names=["ada-002", "mistral-embed", "all-MiniLM-L6-v2","all-distilroberta-v1", 
        "all-mpnet-base-v2", "multi-qa-distilbert-cos-v1"
        ]

datasets=[
    "cnn_dailymail___3.0.0_16384",
    "big_patent___a_16384",
    "EdinburghNLP_-_xsum_16384",
    "gfissore_-_arxiv-abstracts-2021_16384",
    ]

for dataset in datasets:
    for model in model_names:
        with open('./doc_text/embeddings/embeddings_'+dataset+'_'+model+'.pkl', 'wb') as fp:
            pickle.dump(embedding, fp)
```
where embedding is an np.array containing the corresponding text embedding.


Then run the experiment using 
```
poetry run python3 experiment3_doc_experiment.py
poetry run python3 experiment3_doc_plots.py
```

Results then can be visualised via the following notebook

```
experiment3_doc_results.ipynb
```


**Reference:**
Wayland, Jeremy, Corinna Coupette, and Bastian Rieck. "Mapping the Multiverse of Latent Representations." arXiv preprint arXiv:2402.01514. 2024.

## Image Embeddings

It suffices to run the corresponding python script, which automatically downloads CIFAR-10 and creates INCEPTION_V3 embeddings using torchvision. The following script also runs the full experiment:

```
poetry run python3 experiment4_images_dropping.py
```
Results can be visualised in the following notebook
 
```
experiment4_images_results.ipynb
```

**Reference:**
Friedman, Dan, and Adji Bousso Dieng. "The vendi score: A diversity evaluation metric for machine learning." Transactions on Machine Learning Research. 2023.
Krizhevsky, A., Nair, V. and Hinton, G. (2014) The CIFAR-10 Dataset. https://www.cs.toronto.edu/~kriz/cifar.html



## Diversity Evaluation

The following stript contains a class that can be used for all magnitude computations as well as to compute our proposed magnitude-based diversity measures. Feel free to try it out on your own latent representations!
```
from magnipy import Magnipy
```