Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Chain-of-Verification - Prompt Engineering#29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
Ankush-lastmile wants to merge1 commit intohuggingface:main
base:main
Choose a base branch
Loading
fromAnkush-lastmile:pr29

Conversation

Ankush-lastmile
Copy link

Chain-of-Verification - Prompt Engineering

Chain-of-Verification is a prompt engineering technique to reduce hallucinations from LLMs!

Research Paper (Meta AI):https://arxiv.org/pdf/2309.11495.pdf

This recipe uses AIConfig - an open-source config-based framework for building generative AI applications.

https://github.com/lastmile-ai/aiconfig/tree/main

@review-notebook-appReview Notebook App
Copy link

Check out this pull request on ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered byReviewNB

@Ankush-lastmileAnkush-lastmile marked this pull request as ready for reviewFebruary 16, 2024 19:26
@Ankush-lastmile
Copy link
Author

Ankush-lastmile commentedFeb 16, 2024•
edited
Loading

Hi@MKhalusova! Would love a review for this recipe. Thanks!

@@ -0,0 +1,470 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 19, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #2.    import openai

As this is an open source ai cookbook, let's illustrate this technique with anopen source model. From the readme, I can see that these models should be supported:


Reply viaReviewNB

Ankush-lastmile reacted with thumbs up emoji

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've updated the model usage to use TheMixtral-8x7B-Instruct-v0.1, thanks for the suggestion!

MKhalusova reacted with thumbs up emoji
@@ -0,0 +1,470 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 19, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It would be helpful to explain the config a little bit here. At least its essential components. Maybe add some advice on the structure/content of the prompts?


Reply viaReviewNB

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Added a description and links

@@ -0,0 +1,470 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 19, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #3.    verification_question =  "Where was this coding language developed: {{entity}}?"

How do we choose the verification question?


Reply viaReviewNB

@MKhalusova
Copy link
Contributor

MKhalusova commentedFeb 19, 2024•
edited
Loading

A really interesting prompting technique! Thanks for adding! I left some feedback on the notebook, mainly - let's switch to an open source model (I saw that Mistral and Llama are supported), and remove unnecessary TODOs.

In order to publish the notebook, a few more steps are needed:

  • Add the notebook to the_toctree.yml
  • Add the notebook to the list of the latest notebooks in theindex.md. As this is the most recent addition, put it at the top of the list.
  • Right after the first header (notebook title), add yourself as an author, like this:_Authored by: [Your Name](https://huggingface.co/your_profile)_ Feel free to use either your Hugging Face profile, or GitHub profile, it's up to you which one to link.

@MKhalusova
Copy link
Contributor

Once you address this feedback, I'll be able to check if the notebook renders and runs correctly, and if it does, we should be able to merge.

Chain-of-Verification is a prompt engineering technique to reduce hallucinations from LLMs!Research Paper (Meta AI):https://arxiv.org/pdf/2309.11495.pdfThis recipe uses AIConfig - an open-source config-based framework for building generative AI applications.https://github.com/lastmile-ai/aiconfig/tree/main
@Ankush-lastmile
Copy link
Author

Ankush-lastmile commentedFeb 21, 2024•
edited
Loading

  • switched model usage to Mixtral
  • rebase to main
  • removed Todos
  • added descriptions
  • updated index.md &_toctree.yml

@Ankush-lastmile
Copy link
Author

Hi@MKhalusova! Thanks for the review and the detailed comments. I've gone ahead and addressed your comments, as well as made the necessary changes you've requested. It would be great if you could take another look.

Thanks!

@HuggingFaceDocBuilderDev

The docs for this PR livehere. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Minor nit: GitHub (H should be capitalized)


Reply viaReviewNB

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This paragraph can be much shorter:

The next code cell sets up a 'CoVe prompt template' within a structure known as AIConfig. AIConfig is a data format to organize prompt templates and specific model settings. To create a config you can either:

* Generate it using theAIConfig Vscode Extension, or

* Download the confighere and load it with:

config = AIConfigRuntime.load('cove_template_config.json').


Reply viaReviewNB

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #7.          "baseline_prompt": "Name 20 programming languages that were developed in the United States. Include the developer name in parantheses.",

minor, typo: it should be "parentheses"


Reply viaReviewNB

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #1.    # verification_question = "Where was {{entity}} born?"

I think you can remove this comment


Reply viaReviewNB

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #10.        verification_completion = await config.run("verification", params, options=inference_options)

I'm getting an error on this line:

UnwrapError: CalledResult.unwrap() on anErr value: "Missing API key 'OPENAI_API_KEY' in environment"

Reply viaReviewNB

@@ -0,0 +1,490 @@
{
Copy link
Contributor

@MKhalusovaMKhalusovaFeb 21, 2024•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Line #5.    # Display with Markdown

There seems to be an issue rendering this output. Let's switch to a simple print.


Reply viaReviewNB

@MKhalusova
Copy link
Contributor

MKhalusova commentedFeb 21, 2024•
edited
Loading

Thank you for iterating on the cookbook notebook! I found a few typos and minor issues, but there's also an important one - the notebook throws an error:
UnwrapError: CalledResult.unwrap()on anErr value: "Missing API key 'OPENAI_API_KEY' in environment"

Let's make sure it runs smoothly without errors.

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@MKhalusovaMKhalusovaMKhalusova left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

3 participants
@Ankush-lastmile@MKhalusova@HuggingFaceDocBuilderDev
[8]ページ先頭
©2009-2025 Movatter.jp