# Collaborative Computerized Adaptive Testing for Ranking



## Requirements

To install requirements:

```setup
pip install -r requirements.txt
```

## Data Preprocessing

**！！！The preprocessing data is provided in Supplementary_material, which means you can skip this step.**

To get the dataset used in experiments, you can use **[Edudata](https://github.com/bigdata-ustc/EduData)** package.

```train
edudata download NIPS-2020
edudata download junyi
```

Deconstruct the downloaded data as follows:

```train
data/
│
├── NIPS2020/
│   ├── train_task_3_4.csv
│   └── meta_data.csv
│
├── JUNYI/
│   ├── junyi_ProblemLog_original.csv
│   └── junyi_Exercise_table.csv
│
└── dataset.py
│
└── mcmc.py
│
└── prepare_data.py
│
└── setting.py
```

To preprocessing the dataset,run:

```train
cd data
python prepare_data.py --data_name='NIPS2020'
python prepare_data.py --data_name='JUNYI'
```

>📋  **prepare_data.py** will delete students with less than 50 answering records, as well as delete questions with less than 50 answering times. The dataset will be divided into a training set (collaborative students) and a testing set (tested students) in a 4:1 ratio. The outputs of prepare_data.py are **train_triples.csv**, **test_triples.csv**, **triples.csv**, **metadata.json**, **concept_map.json**.

| Dataset                     | NIPS-EDU(NIPS2020) | JUNYI    |
| --------------------------- | ------------------ | -------- |
| \#Students                  | 4,914              | 8,852    |
| \#Questions                 | 900                | 702      |
| \#Response logs             | 1,382,173          | 801,270  |
| \#Response logs per student | 281.27             | 90.52    |
| #Response logs per question | 1,535.75           | 1,141.41 |

 To get the parameter of IRT estimated by mcmc method, run:

```train
python mcmc.py --data_name='NIPS2020'
```

>📋  **mcmc.py** will use Monte Carlo sampling on the dataset to perform posterior estimation on the IRT model, in order to obtain the parameters of the IRT model. The outputs are **alpha.npy** and **beta.npy**, which contains the discrimination and difficulty of questions.

After the data preprocess, the fold becomes(which is provided):

```train
data/
│
├── NIPS2020/
│   ├── alpha.npy
│   ├── beta.npy
│   ├── concept_map.json
│   ├── metadata.json
│   ├── test_triples.csv
│   ├── train_triples.csv
│   └── triples.csv
│
├── JUNYI/
│	├── alpha.npy
│   ├── beta.npy
│   ├── concept_map.json
│   ├── metadata.json
│   ├── test_triples.csv
│   ├── train_triples.csv
│   └── triples.csv
```

To preprocessing the dataset,run:

## Evaluation

To evaluate my model, run:

```train
python main.py --method='ccat' --irt_method='mcmc' --data_name='NIPS2020' --seed=2024 --device='cuda'
```

>📋  main.py contains code for selecting questions and evaluating **ranking consistency**. 
>
>This includes the following hyperparameters:
>
>**method** can choose the selection strategy of CAT, including random, **fsi**, **kli**, **maat**, **ncat**, **becat**, **ccat**. 
>
>**irt_method** controls the training method of the irt model, including **mcmc** and **gd**. 
>
>**data_name** is the dataset used in the experiment, including **NIPS2020** and **JUNYI**. 
>
>**seed** controls the initialization parameters of student abilities, and the seed used in the article is **2023**, **2024**, **2025**, **2026**, **2027**.

## Results

Our model achieves the following performance on :

### Intra Ranking Consistency(MCMC)

| NIPS2020 | Step 5            | Step 10           | Step 15           | Step 20           |
| -------- | ----------------- | ----------------- | ----------------- | ----------------- |
| Random   | 0.7411/0.7531     | 0.8061/0.8084     | 0.8348/0.8363     | 0.8540/0.8547     |
| FSI      | 0.7912/0.7933     | 0.8570/0.8573     | 0.8846/0.8848     | 0.8975/**0.8977** |
| KLI      | 0.7821/0.7839     | 0.8532/0.8530     | 0.8804/0.8805     | 0.8965/0.8966     |
| MAAT     | 0.6762/0.6909     | 0.8083/0.8090     | 0.8588/0.8595     | 0.8843/0.8848     |
| NCAT     | 0.7766/0.7923     | 0.8451/0.8501     | 0.8710/0.8725     | 0.8831/0.8840     |
| BECAT    | 0.7685/0.7680     | 0.8441/0.8449     | 0.8766/0.8771     | 0.8958/0.8961     |
| CCAT     | 0.7982/**0.8149** | 0.8561/**0.8635** | 0.8832/**0.8851** | 0.8955/0.8969     |

>📋  The results in each grid are evaluated using IRT on the left and CCAT on the right, respectively.

### Inter Ranking Consistency

| NIPS2020 | Step 5     | Step 10    | Step 15    | Step 20    |
| -------- | ---------- | ---------- | ---------- | ---------- |
| Random   | 0.7798     | 0.8325     | 0.8590     | 0.8760     |
| FSI      | 0.8258     | 0.8785     | 0.9013     | **0.9126** |
| KLI      | 0.8195     | 0.8758     | 0.8985     | 0.9119     |
| MAAT     | 0.7242     | 0.8373     | 0.8807     | 0.9023     |
| NCAT     | 0.8286     | 0.8697     | 0.8892     | 0.8994     |
| BECAT    | 0.8045     | 0.8676     | 0.8948     | 0.9104     |
| CCAT     | 0.8476**** | **0.8839** | **0.9013** | 0.9116     |

## Contributing

>📋  We use the MIT License.
