- Notifications
You must be signed in to change notification settings - Fork331
QA via natural language AI tests
License
antiwork/shortest
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
AI-powered natural language end-to-end testing framework.
LS27C33xG.mp4
- Natural language E2E testing framework
- AI-powered test execution using Anthropic Claude API
- Built on Playwright
- GitHub integration with 2FA support
- Email validation with Mailosaur
If helpful,here's a short video!
Use theshortest init command to streamline the setup process in a new or existing project.
Theshortest init command will:
npx @antiwork/shortest init
This will:
- Automatically install the
@antiwork/shortestpackage as a dev dependency if it is not already installed - Create a default
shortest.config.tsfile with boilerplate configuration - Generate a
.env.localfile (unless present) with placeholders for required environment variables, such asANTHROPIC_API_KEY - Add
.env.localand.shortest/to.gitignore
- Determine your test entry and add your Anthropic API key in config file:
shortest.config.ts
importtype{ShortestConfig}from"@antiwork/shortest";exportdefault{headless:false,baseUrl:"http://localhost:3000",browser:{contextOptions:{ignoreHTTPSErrors:true},},testPattern:"**/*.test.ts",ai:{provider:"anthropic",},}satisfiesShortestConfig;
The Anthropic API key defaults toSHORTEST_ANTHROPIC_API_KEY /ANTHROPIC_API_KEY environment variables. Can be overwritten viaai.config.apiKey.
Optionally, you can configure browser behavior using thebrowser.contextOptions property in your configuration file. This allows you to pass customPlaywright browser context options.
- Create test files using the pattern specified in the config:
app/login.test.ts
import{shortest}from"@antiwork/shortest";shortest("Login to the app using email and password",{username:process.env.GITHUB_USERNAME,password:process.env.GITHUB_PASSWORD,});
You can also use callback functions to add additional assertions and other logic. AI will execute the callback function after the testexecution in browser is completed.
import{shortest}from"@antiwork/shortest";import{db}from"@/lib/db/drizzle";import{users}from"@/lib/db/schema";import{eq}from"drizzle-orm";shortest("Login to the app using username and password",{username:process.env.USERNAME,password:process.env.PASSWORD,}).after(async({ page})=>{// Get current user's clerk ID from the pageconstclerkId=awaitpage.evaluate(()=>{returnwindow.localStorage.getItem("clerk-user");});if(!clerkId){thrownewError("User not found in database");}// Query the databaseconst[user]=awaitdb.select().from(users).where(eq(users.clerkId,clerkId)).limit(1);expect(user).toBeDefined();});
You can use lifecycle hooks to run code before and after the test.
import{shortest}from"@antiwork/shortest";shortest.beforeAll(async({ page})=>{awaitclerkSetup({frontendApiUrl:process.env.PLAYWRIGHT_TEST_BASE_URL??"http://localhost:3000",});});shortest.beforeEach(async({ page})=>{awaitclerk.signIn({ page,signInParams:{strategy:"email_code",identifier:"iffy+clerk_test@example.com",},});});shortest.afterEach(async({ page})=>{awaitpage.close();});shortest.afterAll(async({ page})=>{awaitclerk.signOut({ page});});
Shortest supports flexible test chaining patterns:
// Sequential test chainshortest(["user can login with email and password","user can modify their account-level refund policy",]);// Reusable test flowsconstloginAsLawyer="login as lawyer with valid credentials";constloginAsContractor="login as contractor with valid credentials";constallAppActions=["send invoice to company","view invoices"];// Combine flows with spread operatorshortest([loginAsLawyer, ...allAppActions]);shortest([loginAsContractor, ...allAppActions]);
Test API endpoints using natural language
constreq=newAPIRequest({baseURL:API_BASE_URI,});shortest("Ensure the response contains only active users",req.fetch({url:"/users",method:"GET",params:newURLSearchParams({active:true,}),}),);
Or simply:
shortest(` Test the API GET endpoint${API_BASE_URI}/users with query parameter { "active": true } Expect the response to contain only active users`);
pnpm shortest# Run all testspnpm shortest login.test.ts# Run specific tests from a filepnpm shortest login.test.ts:23# Run specific test from a file using a line numberpnpm shortest --headless# Run in headless mode using
You can find example tests in theexamples directory.
You can run Shortest in your CI/CD pipeline by running tests in headless mode. Make sure to add your Anthropic API key to your CI/CD pipeline secrets.
Shortest supports login using GitHub 2FA. For GitHub authentication tests:
- Go to your repository settings
- Navigate to "Password and Authentication"
- Click on "Authenticator App"
- Select "Use your authenticator app"
- Click "Setup key" to obtain the OTP secret
- Add the OTP secret to your
.env.localfile or use the Shortest CLI to add it - Enter the 2FA code displayed in your terminal into Github's Authenticator setup page to complete the process
shortest --github-code --secret=<OTP_SECRET>
Required in.env.local:
ANTHROPIC_API_KEY=your_api_keyGITHUB_TOTP_SECRET=your_secret# Only for GitHub auth testsTheNPM package is located inpackages/shortest/. SeeCONTRIBUTING guide.
This guide will help you set up the Shortest web app for local development.
- React >=19.0.0 (if using with Next.js 14+ or Server Actions)
- Next.js >=14.0.0 (if using Server Components/Actions)
Warning
Using this package with React 18 in Next.js 14+ projects may cause type conflicts with Server Actions anduseFormStatus
If you encounter type errors with form actions or React hooks, ensure you're using React 19
- Clone the repository:
git clone https://github.com/antiwork/shortest.gitcd shortest- Install dependencies:
npm install -g pnpmpnpm install
Pull Vercel env vars:
pnpm i -g vercelvercel linkvercel env pull
- Run
pnpm run setupto configure the environment variables. - The setup wizard will ask you for information. Refer to "Services Configuration" section below for more details.
pnpm drizzle-kit generatepnpm db:migratepnpm db:seed# creates stripe products, currently unusedYou'll need to set up the following services for local development. If you're not an Antiwork Vercel team member, you'll need to either run the setup wizardpnpm run setup or manually configure each of these services and add the corresponding environment variables to your.env.local file:
Clerk
- Go toclerk.com and create a new app.
- Name it whatever you like anddisable all login methods except GitHub.

- Once created, copy the environment variables to your
.env.localfile.
- In the Clerk dashboard, disable the "Require the same device and browser" setting to ensure tests with Mailosaur work properly.
Vercel Postgres
- Go to your dashboard atvercel.com.
- Navigate to the Storage tab and click the
Create Databasebutton.
- Choose
Postgresfrom theBrowse Storagemenu.
- Copy your environment variables from the
Quickstart.env.localtab.
Anthropic
- Go to your dashboard atanthropic.com and grab your API Key.
Stripe
- Go to your
Developersdashboard atstripe.com. - Turn on
Test mode. - Go to the
API Keystab and copy yourSecret key.
- Go to the terminal of your project and type
pnpm run stripe:webhooks. It will prompt you to login with a code then give you yourSTRIPE_WEBHOOK_SECRET.
GitHub OAuth
Create a GitHub OAuth App:
- Go to your GitHub account settings.
- Navigate to
Developer settings>OAuth Apps>New OAuth App. - Fill in the application details:
Configure Clerk with GitHub OAuth:
Mailosaur
- Sign up for an account with Mailosaur.
- Create a new Inbox/Server.
- Go toAPI Keys and create a standard key.
- Update the environment variables:
MAILOSAUR_API_KEY: Your API keyMAILOSAUR_SERVER_ID: Your server ID
The email used to test the login flow will have the formatshortest@<MAILOSAUR_SERVER_ID>.mailosaur.net, whereMAILOSAUR_SERVER_ID is your server ID.Make sure to add the email as a new user under the Clerk app.
Run the development server:
pnpm dev
Openhttp://localhost:3000 in your browser to see the app in action.
About
QA via natural language AI tests
Topics
Resources
License
Contributing
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.


