You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
Guidance is available through PyPI and supports a variety of backends (Transformers, llama.cpp, OpenAI, etc.).If you already have the backend required for your model, you can simply run
pip install guidance
Features
A Pythonic interface for language models
When using Guidance, you can work with large language models using common Python idioms:
fromguidanceimportsystem,user,assistant,genfromguidance.modelsimportTransformers# Could also do LlamaCpp or many other modelsphi_lm=Transformers("microsoft/Phi-4-mini-instruct")# Model objects are immutable, so this is a copylm=phi_lmwithsystem():lm+="You are a helpful assistant"withuser():lm+="Hello. What is your name?"withassistant():lm+=gen(max_tokens=20)print(lm)
If run at the command line, this will produce output like:
<|system|>You are a helpful assistant<|end|><|user|>Hello. What is your name?<|end|><|assistant|>I am Phi, an AI developed by Microsoft. How can I help you today?
However, if running in a Jupyter notebook, then Guidance provides a widget for a richer user experience:
With Guidance, it's really easy to capture generated text:
# Get a new copy of the Modellm=phi_lmwithsystem():lm+="You are a helpful assistant"withuser():lm+="Hello. What is your name?"withassistant():lm+=gen(name="lm_response",max_tokens=20)print(f"{lm['lm_response']=}")
lm['lm_response']='I am Phi, an AI developed by Microsoft. How can I help you today?'
Guarantee output syntax with constrained generation
Guidance provides an easy to use, yet immensely powerful syntax for constraining the output of a language model.For example, agen() call can be constrained to match a regular expression:
lm=phi_lmwithsystem():lm+="You are a teenager"withuser():lm+="How old are you?"withassistant():lm+=gen("lm_age",regex=r"\d+",temperature=0.8)print(f"The language model is{lm['lm_age']} years old")
The language model is 13 years old
Often, we know that the output has to be an item from a list we know in advance.Guidance provides aselect() function for this scenario:
fromguidanceimportselectlm=phi_lmwithsystem():lm+="You are a geography expert"withuser():lm+="""What is the capital of Sweden? Answer with the correct letter. A) Helsinki B) Reykjavík C) Stockholm D) Oslo """withassistant():lm+=select(["A","B","C","D"],name="model_selection")print(f"The model selected{lm['model_selection']}")
The model selected C
The constraint system offered by Guidance is extremely powerful.It can ensure that the output conforms to any context free grammar (so long as the backend LLM has full support for Guidance).More on this below.
Create your own Guidance functions
With Guidance, you can create your own Guidance functions which can interact with language models.These are marked using the@guidance decorator.Suppose we wanted to answer lots of multiple choice questions.We could do something like the following:
questions= [ {"question" :"Which state has the northernmost capital?","choices" : ["New South Wales","Northern Territory","Queensland","South Australia","Tasmania","Victoria","Western Australia", ],"answer" :1, }, {"question" :"Which of the following is venomous?","choices" : ["Kangaroo","Koala Bear","Platypus", ],"answer" :2, }]
We can use our decorated function likegen() orselect().Thelanguage_model argument will be filled in for us automatically:
lm=phi_lmwithsystem():lm+="You are a student taking a multiple choice test."formcqinquestions:lm_temp=lm+zero_shot_multiple_choice(question=mcq["question"],choices=mcq["choices"])converted_answer=ord(lm_temp["string_choice"])-ASCII_OFFSETprint(lm_temp)print(f"LM Answer:{converted_answer}, Correct Answer:{mcq['answer']}")
<|system|>You are a student taking a multiple choice test.<|end|><|user|>Which state has the northernmost capital?a : New South Walesb : Northern Territoryc : Queenslandd : South Australiae : Tasmaniaf : Victoriag : Western Australia<|end|><|assistant|>bLM Answer: 1, Correct Answer: 1<|system|>You are a student taking a multiple choice test.<|end|><|user|>Which of the following is venomous?a : Kangaroob : Koala Bearc : Platypus<|end|><|assistant|>cLM Answer: 2, Correct Answer: 2
Guidance functions can be composed, in order to construct a full context free grammar.For example, we can create Guidance functions to build a simple HTML webpage (note that this isnot a full implementation of HTML).We start with a simple function which will generate text which does not contain any HTML tags.The function is marked asstateless to indicate that we intend to use it for composing a grammar:
Next, we come to the function which generates the complete HTML page.We add the HTML start tag, then generate the header, then body, and then append the ending HTML tag:
lm=phi_lmwithsystem():lm+="You are an expert in HTML"withuser():lm+="Create a simple and short web page about your life story."withassistant():lm+=make_html(name="html_text",temperature=0.7)
When running in a Jupyter Notebook so that the widget is active, we get the following output:
Note the varying highlighting of the generation.This is showing another of Guidance's capabilities: fast-forwarding of tokens.The constraints imposed by a grammar often mean that some tokens are known in advance.Guidance doesn't need the model to generate these; instead it can insert them into the generation.This saves forward passes through the model, and hence reduces GPU usage.For example, in the above HTML generation, Guidance always knows the last opening tag.If the last opened tag was<h1> (for example), then as soon as the model generates</, Guidance can fill inh1> without needing the model to perform a forward pass.
Generating JSON
A JSON schema is actually a context free grammar, and hence it can be used to constrain an LLM using Guidance.This is a common enough case that Guidance provides special support for it.A quick sample, based on a Pydantic model:
importjsonfrompydanticimportBaseModel,Fieldfromguidanceimportjsonasgen_jsonclassBloodPressure(BaseModel):systolic:int=Field(gt=300,le=400)diastolic:int=Field(gt=0,le=20)location:str=Field(max_length=50)model_config=dict(extra="forbid")lm=phi_lmwithsystem():lm+="You are a doctor taking a patient's blood pressure taken from their arm"withuser():lm+="Report the blood pressure"withassistant():lm+=gen_json(name="bp",schema=BloodPressure)print(f"{lm['bp']=}")# Use Python's JSON libraryloaded_json=json.loads(lm["bp"])print(json.dumps(loaded_json,indent=4))# Use Pydanticresult=BloodPressure.model_validate_json(lm["bp"])print(result.model_dump_json(indent=8))
Note that the generated blood pressure is not one the model will have seen for a human.When generating JSON, a substantial number of tokens can often be fast-forwarded, due to the structural constraints imposed by the schema.
About
A guidance language for controlling large language models.
