Repository files navigation

This repo is meant to help developers build integrations between tldraw's canvas and AI tools. It contains several resources that can be used to get information out of tldraw (in order to prompt a model) and to create changes in tldraw based on some generated instructions.

• Join theDiscord channel
• Learn more about thetldraw SDK

The best way to get started is to clone this repository and experiment with its example project.

The module is also distributed as an NPM package,@tldraw/ai. It is meant to be used together with thetldraw SDK.

This repository is a pnpm monorepo. It has three parts:

• /package contains the ai module package itself
• /example/client contains the example's frontend (a Vite app)
• /example/worker contains the example's backend (a Cloudflare Worker)

1. Clone this repository.

2. Enablecorepack

corepackenable

3. Install dependencies usingpnpm.

pnpm i

4. Create a/example/.dev.vars file. Add any environment variables required by the server to the/example/.dev.vars file. By default, our example project requires anOpenAI API Key so your/example/.dev.vars file should look something like this:

OPENAI_API_KEY=sk-proj-rest-of-your-keyANY_OTHER_KEY_YOU_ARE_USING=here

If you need to use public-friendly API keys on the frontend, create a/example/.env file and put them there. Seethis guide for more information about environment variables in Vite.

VITE_LEAKABLE_OPENAI_API_KEY=sk-proj-rest-of-your-keyVITE_SOME_PUBLIC_KEY=sk-proj-rest-of-your-key

5. Start the development server.

pnpm run dev

6. Openlocalhost:5173 in your browser.

You can now make any changes you wish to the example project.

You can now make any changes you wish to the example project.

7. Start hacking

There are a few things you can do right away:

• Tweak the example's system prompt at./worker/do/openai/system-prompt.ts.
• Make the example's system capable of creating new shapes.
• Make the example's system capable of creating new events.

See the README inexample/worker/do/openai/README.md for more information on the example's backend.

Note: If you find yourself needing to make changes to the package code, please let us know on thetldraw discord channel. Your changes would be very useful information as we continue to develop the module!

For production, install the@tldraw/ai package in a new repository, such as a fork of tldraw'sVite template. See thetldraw repository for more resources.

Install the@tldraw/ai package from NPM or your package manager of choice.

npm i @tldraw/ai

Use theuseTldrawAi hook in your code.

functionApp(){return(<divstyle={{position:'fixed',inset:0}}><TldrawpersistenceKey="example"><MyPromptUi/></Tldraw></div>)}constTLDRAW_AI_OPTIONS={transforms:[],generate:async({ editor, prompt, signal})=>{// todo, return changesreturn[]},stream:asyncfunction*({ editor, prompt, signal}){// todo, yield each change as it is ready},}functionMyPromptUi(){constai=useTldrawAi(TLDRAW_AI_OPTIONS)return(<divstyle={{position:'fixed',bottom:100,left:16}}><buttononClick={()=>ai.prompt('draw a unicorn')}>Unicorn</button></div>)}

Read on for more tips about using and configuring the hook.

The fundamental unit in tldraw's ai module is the "change". A change is an instruction to the tldraw editor to do one of the following:

• createShape creates a shape
• updateShape updates a shape
• deleteShape deletes a shape
• createBinding creates a binding
• updateBinding updates a binding
• deleteBinding deletes a binding

Seepackage/src/types.ts for more about each change.

Changes should be generated by thegenerate orstream methods of theuseTldrawAi configuration. You can do generate the changes using whichever method you wish, however the expectation is that you will send information to an LLM or other model to generate changes for you.

You may find that models are bad at generating changes directly. In our example project, we communicate with the LLM using a "simplified" format, parsing the response before sending back the actual changes expected by the ai module.

The ai module has support for "transforms" (TldrawAiTransform). This feature allows you to modify the user's prompt (the data extracted from the canvas) and then use those modifications again later when handling changes. These are useful when preparing the data into a format that is easier for an LLM to work with.

Our example project includes several of these. When extracting data from the canvas,SimpleIds transform replaces tldraw's regular ids (which look likeshape:some_long_uuid) with simplified ids (like1 or2). Later, when handling changes, the transform replaces the simplified ids with their original ids (or creates new ones).

To create a transform, extend theTldrawAiTransform class:

exportclassMyCustomTransformextendsTldrawAiTransform{overridetransformPrompt=(prompt:TLAiPrompt)=>{// modify the prompt when it's first created}overridetransformChange=(change:TLAiChange)=>{// modify each change when it's being handled}}

Pass your new class to theuseTldrawAi hook's configuration.

constMY_STATIC_CONFIG:TldrawAiOptions={transforms:[MyCustomTransform],...etc,}

When the user creates a new prompt, the ai module will create a new instance of each transform to be used for that prompt only. This means that you can stash whatever data you wish on the instance. See the examples inexample/client/transforms as a reference.

TheTldrawAiModule class is responsible for

1. Getting information about the current tldraw editor's canvas
2. Incorporating transforms before and after changes are generated
3. Applying "changes" to the tldraw editor's canvas

The package exports a hook,useTldrawAiModule, that creates an instance of theTldrawAiModule class for you to use in React. This class handles tasks such as getting information out of the tldraw canvas and applying changes to the tldraw canvas.

TheuseTldrawAi hook adds an extra layer of convenience around the ai module. This hook handles many of the standard behaviors for you. While we expect to expand this hook to support more configration, you may find it necessary to create your own version of this hook (based on its source code) in order to customize it further.

The hook responds with three methods:prompt,repeat, andcancel.

• prompt accepts either a string or a configuration object withmessages andstream. By default, theprompt method will call your configuration'sgenerate method. Ifstream is true, then it will call your configuration'sstream method.
• cancel will cancel any currently running generation.
• repeat will apply the same changes that were generated last time. This is useful for debugging.

Generate vs. Stream

You don't need to define bothgenerate andstream, though you should define one of them. If you callai.prompt with thestream flag set to true, but don't havestream implemented, then you'll get an error; likewise, if you callai.prompt without thestream flag and withoutgenerate, then you'll get an error. Just be sure to implement one or both.

Static configuration

If you're using theuseTldrawAi hook, we recommend creating a custom hook that passes static options to theuseTldrawAi hook. See theuseTldrawAiExample hook in our example project as a reference.

exportfunctionuseMyCustomAiHook(){constai=useTldrawAi(MY_STATIC_OPTIONS)}constMY_STATIC_OPTIONS:TldrawAiOptions={transforms:[],generate:async({ editor, prompt, signal})=>{// todo, return changesreturn[]},stream:asyncfunction*({ editor, prompt, signal}){// todo, yield each change as it is ready},}

If youmust define the options inside of a React component, it's important that you memoize the options correctly.

exportfunctionMyPromptUi(){constmyMemoizedOptions=useMemo<TldrawAiOptions>(()=>{return{transforms:[],generate:async({ editor, prompt, signal})=>{return[]},stream:asyncfunction*({ editor, prompt, signal}){},}},[])constai=useTldrawAi(myMemoizedOptions)return<etc/>}

Calling the hook

The ai module relies on the tldraw editor at runtime. You should use theuseTldrawAi hook inside of the tldraw editor's React context, or else provide it with the currenteditor instance as a prop.

You can do that via a child component:

functionApp(){return(<divstyle={{position:'fixed',inset:0}}><TldrawpersistenceKey="example"><MyPromptUi/></Tldraw></div>)}functionMyPromptUi(){constai=useMyCustomAiHook()return(<divstyle={{position:'fixed',bottom:100,left:16}}><buttononClick={()=>ai.prompt('draw a unicorn')}>Unicorn</button></div>)}

Or via the Tldraw component'scomponents prop:

constcomponents:TLComponents={InFrontOfTheCanvas:()=>{constai=useMyCustomAiHook()return(<div><buttononClick={()=>ai.prompt('draw a unicorn')}>Unicorn</button></div>)},}functionApp(){return(<divstyle={{position:'fixed',inset:0}}><TldrawpersistenceKey="example"components={components}/></div>)}

If this is inconvenient—or if you like a challenge—you can also pass theeditor as an argument touseTldrawAi. While this involves some "juggling", it may be useful when you wish to place the ai module into a global context or necessary if you need to use it in different parts of your document tree.

functionApp(){const[editor,setEditor]=useState<Editor|null>(null)return(<divstyle={{position:'fixed',inset:0}}><TldrawBranchonEditorMount={setEditor}/>{editor&&<MyPromptUieditor={editor}/>}</div>)}functionTldrawBranch({ onMount}:{onMount:(editor:Editor)=>void}){return(<divstyle={{position:'fixed',inset:0}}><TldrawpersistenceKey="example"onMount={onEditorMount}/></div>)}functionMyPromptUi({ editor}:Editor){constai=useTldrawAi({ editor, ...MY_STATIC_OPTIONS})return(<divstyle={{position:'fixed',bottom:100,left:16}}><buttononClick={()=>ai.prompt('draw a unicorn')}>Unicorn</button></div>)}

This project is provided under the MIT license foundhere. The tldraw SDK is provided under thetldraw license.

Copyright (c) 2024-present tldraw Inc. The tldraw name and logo are trademarks of tldraw. Please see ourtrademark guidelines for info on acceptable usage.

You can find the @tldraw/ai package on npmhere. You can find tldraw on npmhere.

Found a bug? Pleasesubmit an issue.

Have questions, comments or feedback?Join our discord. For the latest news and release notes, visittldraw.dev.

Find us on Twitter/X at@tldraw or email us atmailto:hello@tldraw.com.