- Notifications
You must be signed in to change notification settings - Fork189
Official Repo for "TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding" [ACL 2025 oral]
License
TIGER-AI-Lab/TheoremExplainAgent
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
🌐 Homepage |📖 arXiv |🤗 HuggingFace Dataset |🎥Video Data |
This repo contains the codebase for our paperTheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding
ACL 2025 main (Oral)
TheoremExplainAgent is an AI system that generates long-form Manim videos to visually explain theorems, proving its deep understanding while uncovering reasoning flaws that text alone often hides.
trailer_caption.mp4
- 2025 Jun 24: Paper got selected for Oral presentation (Top 3%).
- 2025 Jun 8: We released our generated video data for researchers to serve as baselines.
- 2025 May 15: Paper accepted to ACL 2025 main conference.
- 2025 Mar 3: Generation code and Evaluation code released. Thanks for the wait!
Skip this section if you just want to try out the code.If you are researchers who just need the baseline videos as baseline comparison, download it here:
wget --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate'https://docs.google.com/uc?export=download&id=18kmzXvbxaFGyJw-g51jnq9m93v_ez4aJ' -O /tmp/gdrive.html&& wget --load-cookies /tmp/cookies.txt -O baseline_videos.zip"https://drive.usercontent.google.com/download?id=18kmzXvbxaFGyJw-g51jnq9m93v_ez4aJ&export=download&confirm=$(sed -rn's/.*name="confirm" value="([^"]+)".*/\\1/p' /tmp/gdrive.html)&uuid=$(sed -rn's/.*name="uuid" value="([^"]+)".*/\\1/p' /tmp/gdrive.html)"&& rm /tmp/gdrive.html /tmp/cookies.txt
Look at theFAQ section in this README doc if you encountered any errors. If that didnt help, create a issue
- Setting up conda environment
conda create --name tea python=3.12.8conda activate teapip install -r requirements.txt
- You may also need to install latex and other dependencies for Manim Community. Look atManim Installation Docs for more details.
# You might need these dependencies if you are using Linux Ubuntu:sudo apt-get install portaudio19-devsudo apt-get install libsdl-pango-dev- Then Download the Kokoro model and voices using the commands to enable TTS service.
mkdir -p models&& wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/kokoro-v0_19.onnx&& wget -P models https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files/voices.bin
- Create
.envbased on.env.template, filling in the environmental variables according to the models you choose to use.SeeLiteLLM for reference.
touch .env
Then open the.env file and edit it with whatever text editor you like.
Your.env file should look like the following:
# OpenAIOPENAI_API_KEY=""# Azure OpenAIAZURE_API_KEY=""AZURE_API_BASE=""AZURE_API_VERSION=""# Google Vertex AIVERTEXAI_PROJECT=""VERTEXAI_LOCATION=""GOOGLE_APPLICATION_CREDENTIALS=""# Google GeminiGEMINI_API_KEY=""...# Kokoro TTS SettingsKOKORO_MODEL_PATH="models/kokoro-v0_19.onnx"KOKORO_VOICES_PATH="models/voices.bin"KOKORO_DEFAULT_VOICE="af"KOKORO_DEFAULT_SPEED="1.0"KOKORO_DEFAULT_LANG="en-us"
Fill in the API keys according to the model you wanted to use.
- Configure Python path. Note that you need to configure the python path to make it work. Otherwise you may encounter import issues (like not being able to import src etc.)
export PYTHONPATH=$(pwd):$PYTHONPATH
- (Optional) To setup RAG, Seehttps://github.com/TIGER-AI-Lab/TheoremExplainAgent?tab=readme-ov-file#generation-with-rag.
Look at theFAQ section in this README doc if you encountered any errors. If that didnt help, create a issue
The model naming follows the LiteLLM convention. For details on how models should be named, please refer to theLiteLLM documentation.
python generate_video.py \ --model"openai/o3-mini" \ --helper_model"openai/o3-mini" \ --output_dir"output/your_exp_name" \ --topic"your_topic" \ --context"description of your topic, e.g. 'This is a topic about the properties of a triangle'" \
Example:
python generate_video.py \ --model"openai/o3-mini" \ --helper_model"openai/o3-mini" \ --output_dir"output/my_exp_name" \ --topic"Big O notation" \ --context"most common type of asymptotic notation in computer science used to measure worst case complexity" \
python generate_video.py \ --model"openai/o3-mini" \ --helper_model"openai/o3-mini" \ --output_dir"output/my_exp_name" \ --theorems_path data/thb_easy/math.json \ --max_scene_concurrency 7 \ --max_topic_concurrency 20 \
Before using RAG, download the RAG documentation from thisGoogle Drive link. After downloading, unzip the file. For example, if you unzip it todata/rag/manim_docs, then you should set--manim_docs_path todata/rag/manim_docs. The vector database will be created the first time you run with RAG.
python generate_video.py \ --model"openai/o3-mini" \ --helper_model"openai/o3-mini" \ --output_dir"output/with_rag/o3-mini/vtutorbench_easy/math" \ --topic"Big O notation" \ --context"most common type of asymptotic notation in computer science used to measure worst case complexity" \ --use_rag \ --chroma_db_path"data/rag/chroma_db" \ --manim_docs_path"data/rag/manim_docs" \ --embedding_model"vertex_ai/text-embedding-005"
We support more options for generation, see below for more details:
usage: generate_video.py [-h] [--model] [--topic TOPIC] [--context CONTEXT] [--helper_model] [--only_gen_vid] [--only_combine] [--peek_existing_videos] [--output_dir OUTPUT_DIR] [--theorems_path THEOREMS_PATH] [--sample_size SAMPLE_SIZE] [--verbose] [--max_retries MAX_RETRIES] [--use_rag] [--use_visual_fix_code] [--chroma_db_path CHROMA_DB_PATH] [--manim_docs_path MANIM_DOCS_PATH] [--embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005}] [--use_context_learning] [--context_learning_path CONTEXT_LEARNING_PATH] [--use_langfuse] [--max_scene_concurrency MAX_SCENE_CONCURRENCY] [--max_topic_concurrency MAX_TOPIC_CONCURRENCY] [--debug_combine_topic DEBUG_COMBINE_TOPIC] [--only_plan] [--check_status] [--only_render] [--scenes SCENES [SCENES ...]]Generate Manim videos using AIoptions:-h, --help show thishelp message andexit --model Select the AI model to use --topic TOPIC Topic to generate videosfor --context CONTEXT Context of the topic --helper_model Select the helper model to use --only_gen_vid Only generate videos to existing plans --only_combine Only combine videos --peek_existing_videos, --peek Peek at existing videos --output_dir OUTPUT_DIR Output directory --theorems_path THEOREMS_PATH Path to theorems json file --sample_size SAMPLE_SIZE, --sample SAMPLE_SIZE Number of theorems to sample --verbose Print verbose output --max_retries MAX_RETRIES Maximum number of retriesfor code generation --use_rag, --rag Use Retrieval Augmented Generation --use_visual_fix_code, --visual_fix_code Use VLM to fix code with rendered visuals --chroma_db_path CHROMA_DB_PATH Path to Chroma DB --manim_docs_path MANIM_DOCS_PATH Path to manim docs --embedding_model {azure/text-embedding-3-large,vertex_ai/text-embedding-005} Select the embedding model to use --use_context_learning Use context learning with example Manim code --context_learning_path CONTEXT_LEARNING_PATH Path to context learning examples --use_langfuse Enable Langfuse logging --max_scene_concurrency MAX_SCENE_CONCURRENCY Maximum number of scenes to process concurrently --max_topic_concurrency MAX_TOPIC_CONCURRENCY Maximum number of topics to process concurrently --debug_combine_topic DEBUG_COMBINE_TOPIC Debug combine videos --only_plan Only generate scene outline and implementation plans --check_status Check planning and code statusfor all theorems --only_render Only render scenes without combining videos --scenes SCENES [SCENES ...] Specific scenes to process (if theorems_path is provided)Note that Gemini and GPT4o is required for evaluation.
Currently, evaluation requires a video file and a subtitle file (SRT format).
Video evaluation:
usage: evaluate.py [-h] [--model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}] [--model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05}] [--model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0}] [--eval_type {text,video,image,all}] --file_path FILE_PATH --output_folder OUTPUT_FOLDER [--retry_limit RETRY_LIMIT] [--combine] [--bulk_evaluate] [--target_fps TARGET_FPS] [--use_parent_folder_as_topic] [--max_workers MAX_WORKERS]Automatic evaluation of theorem explanation videos with LLMsoptions: -h, --help show thishelp message andexit --model_text {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0} Select the AI model to usefor text evaluation --model_video {gemini/gemini-1.5-pro-002,gemini/gemini-2.0-flash-exp,gemini/gemini-2.0-pro-exp-02-05} Select the AI model to usefor video evaluation --model_image {gemini/gemini-1.5-pro-002,gemini/gemini-1.5-flash-002,gemini/gemini-2.0-flash-001,vertex_ai/gemini-1.5-flash-002,vertex_ai/gemini-1.5-pro-002,vertex_ai/gemini-2.0-flash-001,openai/o3-mini,gpt-4o,azure/gpt-4o,azure/gpt-4o-mini,bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0,bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0,bedrock/anthropic.claude-3-5-haiku-20241022-v1:0,bedrock/us.anthropic.claude-3-7-sonnet-20250219-v1:0} Select the AI model to usefor image evaluation --eval_type {text,video,image,all} Type of evaluation to perform --file_path FILE_PATH Path to a file or a theorem folder --output_folder OUTPUT_FOLDER Directory to store the evaluation files --retry_limit RETRY_LIMIT Number of retry attemptsfor each inference --combine Combine all results into a single JSON file --bulk_evaluate Evaluate a folder of theorems together --target_fps TARGET_FPS Target FPSfor video processing. If not set, original video FPS will be used --use_parent_folder_as_topic Use parent folder name as topic namefor single file evaluation --max_workers MAX_WORKERS Maximum number of concurrent workersfor parallel processing- For
file_path, it is recommended to pass a folder containing both an MP4 file and an SRT file.
If you want to modify the system prompt, you need to:
- Modify files in
task_generator/prompts_rawfolder. - Run
task_generator/parse_prompt.pyto rebuild the__init__.pyfile.
cdtask_generatorpythonparse_prompt.pycd ..
TheoremExplainBench can be found onhttps://huggingface.co/datasets/TIGER-Lab/TheoremExplainBench.
How to use:
importdatasetsdataset=datasets.load_dataset("TIGER-Lab/TheoremExplainBench")
Dataset info:
DatasetDict({ train: Dataset({ features: ['uid','subject','difficulty','theorem','description','subfield'], num_rows: 240 })})The FAQ should cover the most common errors you could encounter. If you see something new, report it on issues.
Q: Errorsrc.utils.kokoro_voiceover import KokoroService # You MUST import like this as this is our custom voiceover service. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ModuleNotFoundError: No module named 'src'.
A: Please runexport PYTHONPATH=$(pwd):$PYTHONPATH when you start a new terminal.
Q: ErrorFiles not found
A: Check your Manim installation.
Q: Errorlatex ...
A: Check your latex installation.
Q: The output log is not showing response?
A: It could be API-related issues. Make sure your.env file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues.
Q: Plans / Scenes are missing?
A: It could be API-related issues. Make sure your.env file is properly configured (fill in your API keys), or you can enable litellm debug mode to figure out the issues.
Please kindly cite our paper if you use our code, data, models or results:
@misc{ku2025theoremexplainagentmultimodalexplanationsllm,title={TheoremExplainAgent: Towards Multimodal Explanations for LLM Theorem Understanding},author={Max Ku and Thomas Chong and Jonathan Leung and Krish Shah and Alvin Yu and Wenhu Chen},year={2025},eprint={2502.19400},archivePrefix={arXiv},primaryClass={cs.AI},url={https://arxiv.org/abs/2502.19400}, }
This project is released under thethe MIT License.
We want to thankVotee AI for sponsoring API keys to access the close-sourced models.
The code is built upon the below repositories, we thank all the contributors for open-sourcing.
This work is intended for research purposes only. The authors do not encourage or endorse the use of this codebase for commercial applications. The code is provided "as is" without any warranties, and users assume all responsibility for its use.
Tested Environment: MacOS, Linux
About
Official Repo for "TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understanding" [ACL 2025 oral]
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors5
Uh oh!
There was an error while loading.Please reload this page.