Code/Package

Installation

pip install selfcheckgpt

SelfCheckGPT Usage: BERTScore, QA, n-gram

# Include necessary packages (torch, spacy, ...)fromselfcheckgpt.modeling_selfcheckimportSelfCheckMQAG,SelfCheckBERTScore,SelfCheckNgramdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")selfcheck_mqag=SelfCheckMQAG(device=device)# set device to 'cuda' if GPU is availableselfcheck_bertscore=SelfCheckBERTScore(rescale_with_baseline=True)selfcheck_ngram=SelfCheckNgram(n=1)# n=1 means Unigram, n=2 means Bigram, etc.# LLM's text (e.g. GPT-3 response) to be evaluated at the sentence level  & Split it into sentencespassage="Michael Alan Weiner (born March 31, 1942) is an American radio host. He is the host of The Savage Nation."sentences= [sent.text.strip()forsentinnlp(passage).sents]# spacy sentence tokenizationprint(sentences)['Michael Alan Weiner (born March 31, 1942) is an American radio host.','He is the host of The Savage Nation.']# Other samples generated by the same LLM to perform self-check for consistencysample1="Michael Alan Weiner (born March 31, 1942) is an American radio host. He is the host of The Savage Country."sample2="Michael Alan Weiner (born January 13, 1960) is a Canadian radio host. He works at The New York Times."sample3="Michael Alan Weiner (born March 31, 1942) is an American radio host. He obtained his PhD from MIT."# --------------------------------------------------------------------------------------------------------------- ## SelfCheck-MQAG: Score for each sentence where value is in [0.0, 1.0] and high value means non-factual# Additional params for each scoring_method:# -> counting: AT (answerability threshold, i.e. questions with answerability_score < AT are rejected)# -> bayes: AT, beta1, beta2# -> bayes_with_alpha: beta1, beta2sent_scores_mqag=selfcheck_mqag.predict(sentences=sentences,# list of sentencespassage=passage,# passage (before sentence-split)sampled_passages= [sample1,sample2,sample3],# list of sampled passagesnum_questions_per_sent=5,# number of questions to be drawnscoring_method='bayes_with_alpha',# options = 'counting', 'bayes', 'bayes_with_alpha'beta1=0.8,beta2=0.8,# additional params depending on scoring_method)print(sent_scores_mqag)# [0.30990949 0.42376232]# --------------------------------------------------------------------------------------------------------------- ## SelfCheck-BERTScore: Score for each sentence where value is in [0.0, 1.0] and high value means non-factualsent_scores_bertscore=selfcheck_bertscore.predict(sentences=sentences,# list of sentencessampled_passages= [sample1,sample2,sample3],# list of sampled passages)print(sent_scores_bertscore)# [0.0695562  0.45590915]# --------------------------------------------------------------------------------------------------------------- ## SelfCheck-Ngram: Score at sentence- and document-level where value is in [0.0, +inf) and high value means non-factual# as opposed to SelfCheck-MQAG and SelfCheck-BERTScore, SelfCheck-Ngram's score is not boundedsent_scores_ngram=selfcheck_ngram.predict(sentences=sentences,passage=passage,sampled_passages= [sample1,sample2,sample3],)print(sent_scores_ngram)# {'sent_level': { # sentence-level score similar to MQAG and BERTScore variant#     'avg_neg_logprob': [3.184312, 3.279774],#     'max_neg_logprob': [3.476098, 4.574710]#     },#  'doc_level': {  # document-level score such that avg_neg_logprob is computed over all tokens#     'avg_neg_logprob': 3.218678904916201,#     'avg_max_neg_logprob': 4.025404834169327#     }# }

SelfCheckGPT Usage: NLI (recommended)

fromselfcheckgpt.modeling_selfcheckimportSelfCheckNLIdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")selfcheck_nli=SelfCheckNLI(device=device)# set device to 'cuda' if GPU is availablesent_scores_nli=selfcheck_nli.predict(sentences=sentences,# list of sentencessampled_passages= [sample1,sample2,sample3],# list of sampled passages)print(sent_scores_nli)# [0.334014 0.975106 ] -- based on the example above

SelfCheckGPT Usage: LLM Prompt

# Option1: open-source modelfromselfcheckgpt.modeling_selfcheckimportSelfCheckLLMPromptdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")llm_model="mistralai/Mistral-7B-Instruct-v0.2"selfcheck_prompt=SelfCheckLLMPrompt(llm_model,device)# Option2: API access# (currently only support OpenAI and Groq)# from selfcheckgpt.modeling_selfcheck_apiprompt import SelfCheckAPIPrompt# selfcheck_prompt = SelfCheckAPIPrompt(client_type="openai", model="gpt-3.5-turbo")# selfcheck_prompt = SelfCheckAPIPrompt(client_type="groq", model="llama3-70b-8192", api_key="your-api-key")sent_scores_prompt=selfcheck_prompt.predict(sentences=sentences,# list of sentencessampled_passages= [sample1,sample2,sample3],# list of sampled passagesverbose=True,# whether to show a progress bar)print(sent_scores_prompt)# [0.33333333, 0.66666667] -- based on the example above

Dataset

Update

Option1: HuggingFace

fromdatasetsimportload_datasetdataset=load_dataset("potsawee/wiki_bio_gpt3_hallucination")

Option2: Manual Download

importjsonwithopen("dataset.json","r")asf:content=f.read()dataset=json.loads(content)

Experiments

Probability-based baselines (e.g. GPT-3's probabilities)

Experimental Results

Miscellaneous

MQAG Usage

fromselfcheckgpt.modeling_mqagimportMQAGmqag_model=MQAG()

Acknowledgements

Citation

@article{manakul2023selfcheckgpt,  title={Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models},  author={Manakul, Potsawee and Liusie, Adian and Gales, Mark JF},  journal={arXiv preprint arXiv:2303.08896},  year={2023}}