Repository files navigation

If you find this benchmark helpful, please cite our paper:

@article{xiang2025use,  title={When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation},  author={Xiang, Zhishang and Wu, Chuanjie and Zhang, Qinggang and Chen, Shengyuan and Hong, Zijin and Huang, Xiao and Su, Jinsong},  journal={arXiv preprint arXiv:2506.05690},  year={2025}}

This repository is for the GraphRAG-Bench project, a comprehensive benchmark for evaluating Graph Retrieval-Augmented Generation models.

• [2025-10-27] We releaseLinearRAG, a relation-free method for efficient GraphRAG.
• [2025-08-24] We supportDIGIMON for flexible benchmarking across GraphRAG models.
• [2025-05-25] We releaseGraphRAG-Bench, the benchmark for evaluating GraphRAG models.
• [2025-05-14] We release theGraphRAG-Bench dataset.
• [2025-01-21] We release theGraphRAG survey.

• Introduces Graph Retrieval-Augmented Generation (GraphRAG) concept
• Compares traditional RAG vs GraphRAG approach
• Explains research objective: Identify scenarios where GraphRAG outperforms traditional RAG
• Visual comparison diagram of RAG vs GraphRAG

Two domain-specific leaderboards with comprehensive metrics:

1. GraphRAG-Bench (Novel)

• Evaluates models on literary/fictional content

2. GraphRAG-Bench (Medical)

• Evaluates models on medical/healthcare content

Evaluation Dimensions:

• Fact Retrieval (Accuracy, ROUGE-L)
• Complex Reasoning (Accuracy, ROUGE-L)
• Contextual Summarization (Accuracy, Coverage)
• Creative Generation (Accuracy, Factual Score, Coverage)

Level 1: Fact RetrievalExample: "Which region of France is Mont St. Michel located?"

Level 2: Complex ReasoningExample: "How did Hinze's agreement with Felicia relate to the perception of England's rulers?"

Level 3: Contextual SummarizationExample: "What role does John Curgenven play as a Cornish boatman for visitors exploring this region?"

Level 4: Creative GenerationExample: "Retell King Arthur's comparison to John Curgenven as a newspaper article."

First, install the necessary dependencies for GraphRAG-Bench.

pip install -r requirements.txt

To prevent dependency conflicts, we strongly recommend using separate Conda environments for each framework:

We use the installation of LightRAG as an example. For other frameworks, please refer to their respective installation instructions.

# Create and activate environment (example for LightRAG)conda create -n lightrag python=3.10 -yconda activate lightrag# Install LightRAGgit clone https://github.com/HKUDS/LightRAG.gitcd LightRAGpip install -e.

We provide detailed instructions on how to use GraphRAG-Bench to evaluate each framework.

Specifically, we introduce how to perform index construction and batch inference for each framework in theExamples folder with instructions in theExamples README.

Note that the evaluation code is standardized across all frameworks to ensure fair comparison. Please refer to theEvaluation folder and theEvaluation README for detailed instructions on the evaluation.

Contributions to improve the benchmark website are welcome. Please contact the project team viaGraphRAG@hotmail.com .

If you find this benchmark helpful, please cite our paper:

@article{xiang2025use,  title={When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation},  author={Xiang, Zhishang and Wu, Chuanjie and Zhang, Qinggang and Chen, Shengyuan and Hong, Zijin and Huang, Xiao and Su, Jinsong},  journal={arXiv preprint arXiv:2506.05690},  year={2025}}