Repository files navigation

English |简体中文

YieldLang is ameta-language for generating structured text (ST) that can provide corpora for large language models (LLMs) or guide LLMs to generate ST. Currently provided as aPython package.

• 🧠 Based on a coroutine generator and sampler architecture
• 🤖 Stream-sends characters and parses the context above into a syntax tree
• 🦾 Build formal grammars with classes, methods, and combinators

Work in progress now.

pip install yieldlang

Import theTextGenerator class and define a generator. Thetop method always serves as the entry point for the generator. You can treat the generator as an iterator and use afor loop to iterate over the generated text. For example:

fromyieldlangimportTextGeneratorclassG(TextGenerator):deftop(self):yield"Hello, World!"fortextinG():print(text)

Set another sampler for the generator (default is random sampling). For example, set the large language model sampler:

sampler=MyLLMSampler()print(list(G(sampler)))

Use combinators (e.g.,select,repeat,join, etc.) to define grammar rules in theTextGenerator. For example, for JSON values:

defvalue(self):yieldselect(self.object,self.array,self.string,self.number,self.boolean,self.null    )

This is equivalent to the EBNF form:

value=object|array|string|number|boolean|null

Generate a sequence easily. For example:

defarray(self):yieldselect(        ('[',self.ws,']'),        ('[',self.elements,']')    )

You can get the string just generated and add branches, loops, and other control structures to the generation rules. For example:

defdiagram(self):match (yieldself.diagram_type):case"flowchart":yieldself.flowchartcase"gannt":yieldself.gannt

Use a loop statement in the generator. For example:

defrepeat4(self,s):l:list[str]= []for_inrange(4):l.append((yields))self.do_my_own_thing(l)

Print the generated context tree (convertible to an abstract syntax tree):

defprint_context_tree():ctx=yieldfromG()print(ctx)

For more documentation, please visitdocs.yieldlang.com.

For more information, please refer toCONTRIBUTING.md.

In order forgit to create symbolic links correctly, on Windows you have to run as administrator (Linux users can ignore this):

git clone -c core.symlinks=true https://github.com/YieldLang/yieldlang.git

Install the package in editable mode with the development dependencies:

pip install -e".[dev]"

make run-checks# Run all checks and testsmake build# Build the packagemake docs# Build and watch the docs

Release the YieldLang package. Visit:RELEASE_PROCESS.md

• Guiding Large Language Models to Generate Computer-Parsable Content
  Author:Jiaye Wang Date:2024-03-26 22:54:14

• Python package template atgithub.com/allenai/python-package-template