Repository files navigation

refactor-bot is a CLI for automated code refactoring using OpenAI's LLMs.

Based on the goal provided by the user the CLI will use the OpenAI API togenerate a plan for the refactoring, and refactor one file at a time using "Planand Execute" technique, at the same time making sure the code still compiles andpasses linting and tests.

The difference betweenrefactor-bot and other tools is that it provides richfunctions API based onts-morph that allows it to extract TypeScript specificinformation from the codebase. This deeper insight into the codebase allows theCLI to perform more complex refactoring tasks that span across multiple files.

pnpm add refactor-bot

Don't you want to tinker with the code yourself? Off-course you do. Then you caninstall the CLI from source, and it should be quite easy to do so - no buildprocess is required. It will just run TypeScript.

git clone git@github.com:zaripych/gpt-refactor-bot.git

# Install refactor-bot dependenciespnpm install# Switch back to your project's repository and add refactor-bot as a dependencycd ../your-project-repository pnpm add --save-devfile://../refactor-bot/packages/refactor-bot# Use `tsx` to run refactor-bot TypeScript code directly without building itpnpm add tsx# Run the CLI:pnpm refactor-bot --help

pnpm refactor-bot --help

Create.env file in current directory:

OPENAI_API_KEY="your-key"

Supports commands at the moment:

➜  pnpm refactor-bot<command>Commands:  pnpm refactor-bot prompt      Sends a prompt to the ChatGPT API to generate a                                response  pnpm refactor-bot refactor    Performs a refactoring using Plan and Execute te                                chnique  pnpm refactor-bot completion  generate completion scriptOptions:  --help     Showhelp                                                 [boolean]  --version  Show version number                                       [boolean]

➜  pnpm refactor-bot refactorPerforms a refactoring using Plan and Execute techniqueOptions:  --help         Showhelp                                             [boolean]  --version      Show version number                                   [boolean]  --name         Name of the refactoring to run                         [string]  --id           Unique id of the refactoring that was previously run but didn't                  finish - use this to start from last successful step  [string]  --costs        Whether to print the total costs of OpenAI requests, by default                  it's disabled                       [boolean] [default: false]  --performance  Whether to print performance metrics, by default it's disabled                                                      [boolean] [default: false]

At first it will create a file for you with description of the refactor. Openthe file, edit it in your editor providing as much relevant information as youthink is needed, then rerun the command.

For an example,have a look here.

After creating thegoal description file, we can run the CLI again with thename of the file to start the process.

➜  pnpm refactor-bot refactor --name xx-yy-zz

Refactoring is considered a success if none of the changed files lead toTypeScript or eslint issues and all affected tests pass. Changes which fail thechecks will be reverted.

After refactoring is complete, you will be presented with a report.

Refactoring will not change any files in your local repository and you will beasked to merge the changes yourself.

At the moment the refactoring is performed in a way that doesn't allow moduleinterface to change. This means that the type that represents all exportedmembers of a file will not change as a result of the refactoring. This is doneto ensure that the refactoring is not going to break the codebase. This islikely the most valuable use case, as it is more likely to result in a success.Other options are being considered.

It's likely refactoring might fail for one reason or the other. This couldhappen for following reasons:

• Pre-requisites and expectations about the repository are not met by the CLI,see expectations section below
• The OpenAI model is not capable of performing the refactoring either due tothe model limitations or lack of proper description
• There is a bug in the refactor-bot

In any case, the CLI was built in a way that it can reproduce all the successfulsteps it had done during previous run without starting from scratch, as long aswe know the "refactor-run-id". So if you run the CLI again with the same--idit will start from the last successful step.

UseLOG_LEVEL=debug environment variable to see more information about what'shappening.

For more information about how refactoring works see./DOCUMENTATION.md.

• Source code is TypeScript
• git for version control
• prettier for code formatting
• VSCode as editor (optional)
• You haveglow installed and available in yourPATH for formattingmarkdown in terminals (optional)

Theprompt command allows you to test the functions API we provide to theChatGPT and see what kind of information it can extract from the codebase.

➜  pnpm refactor-bot prompt --watch? Select a file where the conversation is going to be stored › - Use arrow-keys. Return to submit.❯   example-1    example-2    New conversation...

Select a file, and then you will be prompted to enter a message using youreditor. Save the message with--- at the end to send it.

See example conversation at.refactor-bot/prompts/example-1.md.

Therefactor command can do a lot of interesting things. It can be improvedfurther, but there is no way to measure how much we improved it. So the nextstep is to add a way to measure the quality of the refactor for benchmarkingpurposes.

• ability to evaluate result of the refactor to benchmark the quality ofthe refactor, so we can asses how different changes affect the qualityand performance of the refactor-bot
• add Chain Of Thoughts to try to recover from failed attempts to fixissues/tests. at the moment, the algorithm might spent some time in arabbit hole trying to fix eslint issues for changes it could havereverted in the first place
• add evaluation of multiple responses - at the moment we already canintroduce extra iterations to the algorithm, so that instead ofanalyzing just one response from the LLM, we can try to get multiple andassess if one is better than the other - which can be done by runningeslint and tsc on the code generated by the LLM
• add self-verification of the refactor-bot - ie it could try toself-reflect if the results produced by the LLM is good enough andattempt to self-review and try to produce a better output/outcome
• add Tree of Thoughts - we can combine self-assessment capabilities withthe ability to generate multiple responses from the LLM to create atree-like structure of possible outcomes and then evaluate each of themto find the best path forward
• make the algorithm more flexible and allow refactoring process to changethe codebase in a way that is not limited to the current moduleinterface
• spike for an ability to generate a refactor script by LLM which could bepurposefully built and fine-tuned for a particular task - this wouldallow to perform more complex refactoring tasks with higher level taskdescriptions - ie "migrate RxJS from version 6 to version 7" or "migrateto Next.js", etc.
• spike for an ability to debug the code - ie it could then run the userscode in a sandbox and answer questions about the behavior of the code
• spike for an ability to generate new code - for example unit tests, ornew features
• spike for an ability to review the code in a PR and possibly debug someof it, if it looks sketchy, or suggest improvements (which it would havealready tested, linted and debugged for you)
• ability to create pull requests in GitHub for both successfulrefactoring and discarded commits with issues

If you are concerned about privacy, consider using "business" account withOpenAI. Read their license agreement to understand how they can use the data wesend there.

The use of the provided tools here is fully at your own risk. Ensure there areno secrets available to the bot in your projects directory or in environmentvariables so nothing can be leaked accidentally.

Just a minimum precautions have been made so far to safeguard from accidentalleaks:

• We create a temporary directory in$TMPDIR and run all the code againstthe copy, the code is copied from your current repository ignoringeverything from '.gitignore' file so only the source code should be copiedover
• We ensure there are no symlinks leading outside of the temporary directory

We usets-morph to get access to the source code, which usesTypeScriptcompiler, which can read files in the repository. So if your source code has anysecrets directly in code - they might end up being sent to OpenAI API.

The library has capability to run custom code generated by LLM's. At this momentit is only allowed to write TypeScript scripts, which are validated by rollupbundling procedure and allowed to access onlyts-morph library. The use casefor the interpreter is to allow the LLM to generate code which can performadvanced analytics or aggregation usingts-morph API. For an example, see.refactor-bot/prompts/example-5.md.

This capability is experimental and not exposed to therefactor command, butis available whenprompt command is used.

We do not use other external services at the moment other than OpenAI API.