Repository files navigation

Oscar is a project aiming to improve open-source software developmentby creating automated help, or “agents,” for open-source maintenance.We believe there are many opportunities to reduce theamount of toil involved with maintaining open-source projectsboth large and small.

The ability of large language models (LLMs) to do semantic analysis ofnatural language (such as issue reports or maintainer instructions)and to convert between natural language instructions and program codecreates new opportunities for agents to interact more smoothly with people.LLMs will likely end up being only a small (but critical!) part of the picture;the bulk of an agent's actions will be executing standard, deterministic code.

Oscar differs from many development-focused uses of LLMs by not tryingto augment or displace the code writing process at all.After all, writing code is the fun part of writing software.Instead, the idea is to focus on the not-fun parts, like processing incoming issues,matching questions to existing documentation, and so on.

Oscar is very much an experiment. We don't know yet where it will go or whatwe will learn. Even so, our first prototype,the@gabyhelp bot, has already had manysuccessful interactions in the Go issue tracker.

For now, Oscar is being developed under the auspices of the Go project.At some point in the future it may (or may not) be spun out into a separate project.

The rest of this README explains Oscar in more detail.

The concrete goals for the Oscar project are:

• Reduce maintainer effort to resolve issues[note that resolve does not always mean fix]
• Reduce maintainer effort to resolve change lists (CLs) or pull requests (PRs)[note that resolve does not always mean submit/merge]
• Reduce maintainer effort to resolve forum questions
• Enable more people to become productive maintainers

It is a non-goal to automate away coding.Instead we are focused on automating away maintainer toil.

Maintainer toil is not unique to the Go project, so we are aiming to buildan architecture that any software project can reuse and extend,building their own agents customized to their project's needs.Hence Oscar:open-source contributor agent architecture.Exactly what that will mean is still something we are exploring.

So far, we have identified three capabilities that will be an important partof Oscar:

1. Indexing and surfacing related project context duringcontributor interactions.
2. Using natural language to control deterministic tools.
3. Analyzing issue reports, CLs/PRs and group discussions,to help improve them in real time during or shortly after submission,and to label and route them appropriately.

It should make sense that LLMs have something to offer here,because open-source maintenance is fundamentally aboutinteracting with people using natural language, andnatural language is what LLMs are best at.So it's not surprising that all of these have an LLM-related component.On the other hand, all of these are also backed bysignificant amounts of deterministic code.Our approach is to use LLMs for what they're good at—semanticanalysis of natural language and translation fromnatural language into programs—andrely on deterministic code to do the rest.

The following sections look at each of those three important capabilities in turn.Note that we are still experimenting,and we expect to identify additional important capabilities as time goes on.

Software projects are complex beasts.Only at the very beginning can a maintainer expectto keep all the important details and context in their head,and even when that's possible, those being in one person'shead does not help when a new contributor arrives witha bug report, a feature request, or a question.To address this, maintainers write design documentation,API references, FAQs, manual pages, blog posts, and so on.Now, instead of providing context directly, a maintainer canprovide links to written context that already exists.Serving as a project search engine is still not the best use ofthe maintainer's time.Once a project grows even to modest size, any single maintainercannot keep track of all the context that might be relevant,making it even harder to serve as a project search engine.

On the other hand, LLMs turn out to be a great platform forbuilding a project search engine.LLMs can analyze documents and produceembeddings,which are high-dimensional (for example, 768-dimensional)floating point unit vectors with the property that documentswith similar semantic meaning are mapped to vectors that point in similar directions.(For more about embeddings, seethis blog post.)Combined with a vector database to retrieve vectors similarto an input vector,LLM embeddings provide a very effective way to indexall of an open-source project's context, includingdocumentation, issue reports, and CLs/PRs, and forum discussions.When a new issue report arrives, an agent can use the LLM-basedproject context index to identify highly related context,such as similar previous issues or relevant project documentation.

Our prototype agent implements this functionality and replies tonew issues in the Go repository with a list of at most tenhighly related links that add context to the report.(If the agent cannot find anything that looks related enough,it stays quiet and does not reply at all.)In the first few weeks we ran the agent, we identified the followingbenefits of such an agent:

1. The agent surfaces related context to contributors.

   It is common for new issue reports to duplicate existing issue reports:a new bug might be reported multiple times in a short time window,or a non-bug might be reported every few months.When an agent replies with a link to a duplicate report,the contributor can close their new report and then watch that earlier issue.When an agent replies with a link to a report that looks like a duplicatebut is not, the contributor can provide added context to distinguish theirreport from the earlier one.

   For example, ingolang/go#68196,after the agent replied with a near duplicate, the original reporter commented:

   Good bot :). Based on the discussion in this issue, I understand thatit might not be possible to do what's being suggested here.If that's the case I'd still suggest to leave the issue open for a bitto see how many Go users care about this problem.

   As another example, ongolang/go#67986,after the agent replied with an exact duplicate, the original reporter commented:

   Drats, I spent quite a bit of time searching existing issues. Not sure how I missed [that one].

2. The agent surfaces related context even to project maintainers.

   Once a project reaches even modest size, no one person can remember all the context,not even a highly dedicated project maintainer.When an agent replies with a link to a related report,that eliminates the time the maintainer must spend to find it.If the maintainer has forgotten the related report entirely,or never saw it in the first place (perhaps it was handled by someone else),the reply is even more helpful, because it can point the maintainerin the right direction and save them the effort of repeating theanalysis done in the earlier issue.

   For example, ingolang/go#68183,a project maintainer filed abug against the Go compiler for mishandling certain malformed identifiers.The agent replied with a link to a similar report of the same bug,filed almost four years earlier but triaged to low priority.The added context allowed closing the earlier bug andprovided an argument for raising the priority of the new bug.

   As another example, ingolang/go#67938,a project maintainer filed a bug against the Go coverage toolfor causing the compiler to report incorrect sub-line position information.The agent replied with an earlier related issue (incorrect line numbers)from a decade earlieras well as a more recent issue about coveragenot reporting sub-line position information at all.The first bug was important context,and the second bug's “fix” was the root cause of the bug in the new report:the sub-line position information added then was not added correctly.Those links pinpointed the exact code where the bug was.Once that was identified, it was also easy to determine the fix.

3. The agent interacts with bug reporters immediately.

   In all of the previous examples, the fact that the agent replied only a minute or twoafter the report was filed meant that the reporter was still available and engagedenough to respond in a meaningful way: adding details to clarify the suggestion,closing the report as a duplicate, raising bug priority based on past reports,or identifying a fix.In contrast, if hours or days (or more) go by after the initial report,the original reporter may no longer be available, interested, or ableto provide context or additional details.Immediately after the bug report is the best time to engage the reporterand refine the report.Maintainers cannot be expected to be engaged in this work all the time,but an agent can.

Finally, note that surfacing project context is extensible,so that projects can incorporate their context no matter what form it takes.Our prototype's context sources are tailored to the Go project,reading issues from GitHub, documentation fromgo.dev,code reviews from Gerrit, and discussions from Google Groups likegolang-nuts,but the architecture makes it easy to add additional sources.

The second important agent capability is using naturallanguage to control deterministic tooling.As open-source projects grow, the number of helpful tools increases,and it can be difficult to keep track of all of them and rememberhow to use each one.For example, our prototype includes a general facilityfor editing GitHub issue comments to add or fix links.We envision also adding facilities for adding labels toan issue or assigning or CC'ing peoplewhen it matches certain criteria.If a maintainer does not know this functionality existsit might be difficult to find.And even if they know it exists, perhaps they aren't familiarwith the specific API and don't want to take the time to learn it.

On the other hand, LLMs are very good at translating betweenintentions written in natural languageand executable forms of those intentions such as program codeor tool invocations.We have done preliminary experiments with Gemini selecting fromand invoking available tools to satisfy natural language requestsmade by a maintainer.We don't have anything running for real yet,but it looks like a promising approach.

A different approach would be to rely more heavily on LLMs,letting them edit code, issues, and so on entirely based onnatural language prompts with no deterministic tools.This “magic wand” approach demands more of LLMs than theyare capable of today.We believe it will be far more effective to use LLMs to convertfrom natural language to deterministic tool use onceand then apply those deterministic tools automatically.Our approach also limits the amount of “LLM supervision” needed:a person can check that the tool invocation is correctand then rely on the tool to operate deterministically.

We have not built this part of Oscar yet, but when we do,it will be extensible, so that projects can easily plug in their own tools.

The third important agent capability is analyzing issue reportsand CLs/PRs (change lists / pull requests).Posting about related issues is a limited form of analysis,but we plan to add other kinds of semantic analysis,such as determining that an issue is primarily about performanceand should have a “performance” label added.

We also plan to explore whether it is possible to analyze reportswell enough to identify whether more information is needed tomake the report useful. For example, if a report does not includea link to a reproduction program on theGo playground,the agent could ask for one.And if there is such a link, the agent could make sure to inline the codeinto the report to make it self-contained.The agent could potentially also run a sandboxed execution toolto identify which Go releases contain the bug and even usegit bisectto identify the commit that introduced the bug.

As discussed earlier, all of these analyses and resulting interactionswork much better when they happen immediately after the reportis filed, when the reporter is still available and engaged.Automated agents can be on duty 24/7.

We have not built this part of Oscar yet, but when we do,it too will be extensible, so that projects can easily define their ownanalyses customized to the reports they receive.

Our first prototype, Gaby ("Go AI bot"), explores the potential ofopen-source contributor agents. Gaby operates withintheGo issue tracker,posting as@gabyhelp.The source code is ininternal/gaby in this repository.Thegaby package's documentationexplains the overall structure of the code in the repository as well.

Currently, Gaby indexes content from various sources:

• Go issues: GitHub issues for the Go projects.
• Go documentation: Official documentation ongo.dev.
• Code reviews: Reviews fromGerrit.
• Discussions: discussions for the Go project and golang-nuts Google Group.

This allows Gaby to respond to new issues with relevant links and information.

Gaby's structure makes it easy to run on any kind of hosting service,using any LLM, any storage layer, and any vector database.Our current prototype runs onGCP Cloud Runusing Google's Gemini LLM, withGoogle Cloud Firestoreserving as both the key-value storage and vector database.However, as demonstrated in our initial proof-of-concept, Gaby can also functionon local workstation usingPebble andan in-memory vector database.

Running on hosted platforms with dedicated URLs(as opposed to a local workstation)enables integration withGitHub webhooks.This allows for real-time responses to issues and facilitates ongoingconversations.

Our experience with all of this will inform the eventual generalized Oscar design.

There is much work left to do.

The Go project has run its own completely deterministic agent,@gopherbot, for many years.That agent is configured by writing, reviewing, and checking in Go code in thegolang.org/x/build/cmd/gopherbotpackage.Having the agent has been an incredible help to the Go projectand is part of the inspiration for Oscar.At the same time, we are aiming for an even lighter-weightway to configure new agent behaviors: using natural languageto control general behaviors.Over time, our goal is to merge @gabyhelp back into @gopherbotby re-building @gopherbot as an Oscar agent.

We are excited about the opportunities here, but we recognizethat we may be missing important concerns as well asimportant opportunities to reduce open-source maintainer toil.We have createdthis GitHub discussion to discussboth concerns and new ideas for ways that Oscar-based agent canhelp improve open-source maintenance.Feedback there is much appreciated.