Sushant Gautam's submission for IN5550 final exam: Fact-checking track

Helper files

Helper for fine-tuning and evaluating the fact-checker model

usage:pythonfine_tune_hf.py [-h] [--batch_sizeBATCH_SIZE] [--lrLR] [--modelMODEL] [--epochsEPOCHS] [--freezeFREEZE] [--dbpedia_pathDBPEDIA_PATH]                   [--data_pathDATA_PATH] [--plot_roc]optionalarguments:-h,--helpshowthishelpmessageandexit--batch_sizeBATCH_SIZE--lrLR--modelMODEL--epochsEPOCHS--dbpedia_pathDBPEDIA_PATH--data_pathDATA_PATH--plot_rocIfset,theROCcurvewillbeplottedandsaved.

Helper function for LLM-based fact-checking

usage:pythonllm_fact_check.py [-h] [--data_pathDATA_PATH] [--dbpedia_pathDBPEDIA_PATH] [--evidence_pathEVIDENCE_PATH] [--set {test,train,val}]                          [--num_procNUM_PROC] [--llm_knowledge] [--vllm_urlVLLM_URL]optionalarguments:-h,--helpshowthishelpmessageandexit--data_pathDATA_PATH--dbpedia_pathDBPEDIA_PATH--evidence_pathEVIDENCE_PATHPathtotheedvidenceJSONspredictedbyLLM.--set {test,train,val}--num_procNUM_PROC--llm_knowledgeIfset,theinstructionwillbeclaimonlyLLMbasedfactchecking.--vllm_urlVLLM_URLURLofthevLLMserver,e.g.,http://g002:8000

Set up Llama3-Instruct inference server using vLLM

Note: The output of graph filtering using LLM is made available in the repo already. So you can choose to skip inference server set-up and jump to fine-tuning pre-trained models and evaluating them.

Run on any server with NVIDIA A100 GPU (80 GB VRAM)

Request access to the model athttps://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct.

Make sure you have logged in to Hugging Face withhuggingface-cli whoami and have access to the model.

python -m vllm.entrypoints.openai.api_server  --model meta-llama/Meta-Llama-3-8B-Instruct

1. Evaluating Zero-shot Claim Only Baseline with LLM

python llm_fact_check.py --settest --llm_knowledge --vllm_url http://g002:8000

2. Fine-tune and evaluate RoBERTa as Claim Only Baseline

python fine_tune_hf.py --model roberta-base --batch_size 32

3. Filtering connections and caching data for fine-tuning and evaluation

3.1: Filtering Possible Connections with LLM

python llm_filter_relation.py --set train --vllm_url http://g002:8000python llm_filter_relation.py --set val --vllm_url http://g002:8000python llm_filter_relation.py --settest --vllm_url http://g002:8000

unzip llm_v1_jsons.zip

3.2: Fuzzy Relation Mining

3.2.1: Two-stage Fuzzy Relation Mining

python mine_llm_filtered_relation.py --set train --outputPath ./llm_v1/ --jsons_path ./llm_v1_jsons/python mine_llm_filtered_relation.py --set val --outputPath ./llm_v1/ --jsons_path ./llm_v1_jsons/python mine_llm_filtered_relation.py --settest --outputPath ./llm_v1/ --jsons_path ./llm_v1_jsons/

3.2.1: Single-stage Fuzzy Relation Mining

python mine_llm_filtered_relation.py --set train --outputPath ./llm_v1_singleStage/ --jsons_path ./llm_v1_jsons/ --skip_second_stage python mine_llm_filtered_relation.py --set val --outputPath ./llm_v1_singleStage/ --jsons_path ./llm_v1_jsons/ --skip_second_stagepython mine_llm_filtered_relation.py --settest --outputPath ./llm_v1_singleStage/ --jsons_path ./llm_v1_jsons/ --skip_second_stage

4. Zero-shot LLM as Fact Classifier with evidence

python llm_fact_check.py --settest --vllm_url http://g002:8000  --evidence_path ./llm_v1_jsons

5. Fine-tuning pre-trained models

5.1: Fine-tuning Two-stage BERT classifier on the filtered data

python fine_tune_hf.py --model bert-base-uncased --batch_size 64 --data_path ./llm_v1/

5.2: Fine-tuning Single-stage RoBERTa classifier on the filtered data

python fine_tune_hf.py --model roberta-base --batch_size 32 --data_path ./llm_v1_singleStage/

5.3: Fine-tuning Two-stage RoBERTa classifier on the filtered data

python fine_tune_hf.py --model roberta-base --batch_size 32 --data_path ./llm_v1/