# RAGChecker



## 🚀 Quick Start

### Setup Environment

```bash
git clone https://github.com/amazon-science/RefChecker.git
cd RefChecker
git checkout dev
pip install -e .
python -m spacy download en_core_web_sm
pip install rank-eval
```



## Benchmark Datasets

Use the following script to download the corpus and queries for the 10 datasets:
```bash
cd rag_baselines
bash scripts/download_data.sh
```

## Baseline RAG Inference

Run the following command and the example scripts to collect responses from the baseline RAG systems.
```
cd rag_baselines
bash scripts/run_indexing.sh  # Index the documents
bash scripts/run_retrieval.sh  # Retrieve the documents
bash scripts/run_generation.sh  # Generate responses
```

Please modify the environment variables in the scripts to match your setup.

## Checking

If you are using AWS Bedrock version of Llama3 for the claim extractor and checker, use the following command:


```bash
python checking.py \
    --input_path=<path_to_generated_responses> \
    --output_path=<path_to_checking_results> \
    --extractor_name=bedrock/meta.llama3-70b-instruct-v1:0 \
    --checker_name=bedrock/meta.llama3-70b-instruct-v1:0 \
    --batch_size_extractor=64 \
    --batch_size_checker=128 \
    --answer2response \
    --response2answer \
    --retrieved2response \
    --retrieved2answer
```


# Computing Metrics

Use the following command for computing the metrics:

```bash
python rag_eval.py --file=<path_to_checking_results_file>
```