Repository files navigation

EvoMaster (www.evomaster.org) is the first (2016) open-source AI-driven toolthat automaticallygenerates system-level test casesfor web/enterprise applications.This is related toFuzzing.In particular,EvoMaster can fuzz APIs such as REST, GraphQL and RPC.Not onlyEvoMaster can generate inputs that find program crashes, but also it generates small effective test suites (e.g., in Python, JavaScript and Java/Kotlin JUnit format) that can be used forregression testing.

EvoMaster is an AI driven tool.In particular, internally it uses anEvolutionary AlgorithmandDynamic Program Analysis to beable to generate effective test cases.The approach is toevolve test cases from an initial population ofrandom ones, trying to maximize measures like code coverage and fault detection.EvoMaster uses several kinds of AI heuristics to improve performance even further,building on decades of research in the field ofSearch-Based Software Testing.

On a console, copy&paste the following (requiresDocker installed).It will fuzz the PetClinic example API from Swagger, for 30 seconds, as shown in the following video.

docker run -v "$(pwd)/generated_tests":/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

Note that, depending on which shell and operating system you are using, you might need slightly different commands when mounting folders with the-v option.

For example, if run in a MSYS shell on Windows likeGit Bash, there is the need of an extra / before the $.

docker run -v "/$(pwd)/generated_tests":/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

If you are rather using a Command Prompt (Cmd.exe) terminal, you need to use%CD% instead of$(pwd) to refer to the current folder:

docker run -v %CD%/generated_tests:/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

On the other hand, on a PowerShell you need${PWD}:

docker run -v ${PWD}/generated_tests:/generated_tests webfuzzing/evomaster  --blackBox true --maxTime 30s  --ratePerMinute 60 --bbSwaggerUrl  https://petstore.swagger.io/v2/swagger.json

If you encounter issues running the command:

• Ensure Docker is installed and running.
• Check that you have the correct rights/permissions to mount the specified volume.
• Consult the Docker documentation for your shell environment for specific syntax requirements.

Once the command is executed, you can inspect the generated files undergenerated_tests folder.

Note, since version 4.0.0, nowEvoMaster by default also creates an interactive web report.

• Web APIs: At the moment,EvoMaster can generate test cases forREST,GraphQL andRPC (e.g.,gRPC andThrift) APIs.

• Free: this is an open-source project, with funding from public research (e.g., professors, postdocs and PhD students).There is no monetary cost involved in using this tool, or need to use any paid external services (e.g., external LLM APIs).If you are a resident in EU, Norway, Argentina, China, or any associate country involved with ERC, thanks for your tax money supporting this project.

• In-house, no telemetry: currently there is no telemetry in use inEvoMaster, and it does not require to connect to any external service on internet, besides the tested application. AFAIK, it can be run in-house without any worry of leaking any IP to external parties (if not, please open a new issue to tell us how that could happen, and we will try to fix it).

• Black-Box testing mode: can run on any API (regardless of its programming language, e.g., Python and Go).However, results for black-box testing will be worse than white-box testing (e.g., due to lack of code analysis).Default test case output is in Python, but other formats are available as well.

• White-Box testing mode: can be used for APIs compiled toJVM (e.g., Java and Kotlin).EvoMaster analyses the bytecode of the tested applications, and usesseveral heuristics such astestability transformations andtaint analysis to be able to generatemore effective test cases. We support JDK8 and the major LTS versions after that (currently JDK21). Might work on other JVM versions, but we provideNO support for it.Note: there was initial support for other languages as well, like for example JavaScript/TypeScript and C#, but they were not in a stable, feature-complete state. The support for those languages for white-box testing has been dropped, at least for the time being.

• Installation: we provide installers for the main operating systems:Windows (.msi),OSX (.dmg) andLinux (.deb). We also provide an uber-fat JAR file.To download them, see theRelease page.Release notes are present in the filerelease_notes.md.If you are using the uber-fat JAR, it should work with any major LTS version (from JDK 8 on).Whereas for the client library, needed for white-box testing, we will support JDK 8 likely for a long, long while, be warned that future versions of the executable JAR might start to require higher versions of the JDK in a non-so-distant future.If that is going to be higher than your current version of the JVM, if you cannot upgrade or have 2 different JDKs on your machine, then you should not use the uber-jar but rather one of the installers.When you use one of the installers, keep in mind that currently they do not update thePATH variable. This needs to be done manually,see documentation.Also keep in mind we have not paid theMicrosoft/Apple Tax. This means that your operating system by default will block the installation, stating it cannot verify it is not a malware.But the blockcan be bypassed.

• Docker:EvoMaster is now released via Docker as well, underwebfuzzing/evomaster on Docker Hub. For more information on how to useEvoMaster via Docker,see documentation.

• GitHub Action: it is possible to runEvoMaster in GitHub Actions, as part of Continuous Integration, by using thefollowing custom action (which is in a different GitHub repository).

• Hardware: although state-of-the-art AI techniques are used, there is no major hardware requirement to be able to runEvoMaster. It will work even on old laptops. The main computational bottleneck is running the tested applications, and making network calls to them.

• State-of-the-art: anindependent study (2022), comparing 10 fuzzers on 20 RESTful APIs, shows thatEvoMaster gives the best results. Anotherindependent study (2024) done by a different research group confirms these results.

• Schema: REST APIs must provide a schema inOpenAPI/Swaggerformat (eitherv2 orv3).

• Output: the tool generatesJUnit (version 4 or 5) tests, written in either Java or Kotlin, as well as test suites in Python and JavaScript. For a complete list, see the documentation for the CLI parameter--outputFormat.Some examples are: PYTHON_UNITTEST, KOTLIN_JUNIT_5, JAVA_JUNIT_4 and JS_JEST.Note that the generated tests rely on third-party libraries (e.g., to make HTTP calls).These will need to be setup in your projects,see documentation.

• Web Report: besides generating executable tests in different programming language, an interactiveindex.html web report is created as well by default, visualizing and summarizing the results of the generated tests.

• Fault detection:EvoMaster can generate tests cases that reveal faults/bugs in the tested applications.Different heuristics are employed, like checking for 500 status codes, mismatches from the API schemas and access policy violations.

• Self-contained tests: for white-box testing, the generated tests do start/stop the application, binding to an ephemeral port.This means that the generated tests can be used forregression testing (e.g., added to the Git repositoryof the application, and run with any build tool such as Maven and Gradle).For black-box testing, you will need to make sure the application is up and running before executing the tests.

• Database handling: for white-box testing,EvoMaster can intercept and analyse all communications done with SQL and MongoDB databases, and usesuch information to generate higher code coverage test cases. Furthermore, it can generate data directlyinto the databases, and have such initialization automatically added in the generated tests.At the moment,EvoMaster supportsPostgres,MySQL,H2 and MongoDB databases.

• Authentication: we support auth based on authentication headers and cookies.Besides using fixed HTTP headers,it is also possible to declaratively specify which login endpoint should be used to dynamically obtain authentication info (e.g., auth tokens or cookies) for each test execution.See documentation.

• Driver: to be used forwhite-box testing, users need to write adriver manually.We recommend to tryblack-box mode first (should just need a few minutes to get it up and running) to getan idea of whatEvoMaster can do for you.

• JDK 9+: white-box testing requires bytecode manipulation.Each new release of the JDK makes doing this harder and harder.Dealing with JDKs above8 is doable, but it requires some settings.See documentation.

• Execution time: to get good results, you might need to run the search for several hours.We recommend to first try the search for 10 minutes, just to get an idea of what type of tests can be generated.But, then, you should runEvoMaster for something like between 1 and 24 hours (the longer the better, butit is unlikely to get better results after 24 hours).

• RPC APIs: for the moment, we do not directly support RPC schema definitions. Fuzzing RPC APIs requires to write a driver, using the client library of the API to make the calls.

• External services: (e.g., other RESTful APIs) currently there is no support for them (e.g., to automatically mock them).It is work in progress.

• Failing tests: the tests generated byEvoMaster should all pass, and not fail, even when they detect a fault.In those cases, comments/test-names would point out that a test is revealing a possible fault, while still passing.However, in some cases the generated tests might fail. This is due to the so calledflaky tests, e.g., whena test has assertions based on the time clock (e.g., dates and timestamps).There is ongoing effort to address this problem, but it is still not fully solved.

Several enterprises useEvoMaster to fuzz their Web APIs.We do few academia-industry collaborations (see more info here), where we help test engineers to applyEvoMaster on their systems, as long as we can then report on such experience.Examples of Fortune 500 companies usingEvoMaster are:

• Meituan: seeTOSEM'23,ASE'24,SCP'25.

• Volkswagen: seeAUSE'24,ICST'25.

• A45-minute talk given at TestCon'25 on Fuzz Testing Web APIs gives an overview of what can be expected from this kind of fuzzers.Ashorter version (16 minutes)was given at Nordic APIs 2025 Platform Summit.

• Ashort video (5 minutes)shows the use ofEvoMaster on one of thecase studies inEMB.

• This13-minute videoshows how to write a white-box driver for EvoMaster, for therest-api-example.

• How toDownload and Install EvoMaster on Windows 10, using its.msi installer.

• Short presentation (5 minutes) about version 2.0.0.

• Demonstration of Docker and GitHub Actions support.

In the last few years, several few tools have been proposed in the academic literature and in the open-source community.You can read more details in this2023 survey on REST API testing.

Existing open-source tools for REST API fuzzing, with at least 100 stars on GitHub, are for example (in alphabetic order):CATS,Dredd,Fuzz-lightyear,ResTest,Restler,andSchemathesis.

All these tools areblack-box, i.e., they do not analyze the source-code of the tested APIs to generate more effective test data.As we are the authors of EvoMaster, we are too biased to compare it properly with those other black-box tools.However, different independent studies (e.g., in2022 and2024) shows that EvoMaster is among the best performant.Furthermore, if your APIs are running on the JVM (e.g., written in Java or Kotlin), then EvoMaster has clearly an advantage, as it supportswhite-box testing.

If you are trying to useEvoMaster, but the instructions in this documentation are not enough to get you started, or they are too unclear, then it means it is abug in the documentation, which then would need to be clarified and updated. In such cases, please create a newissue.

Also, feel free to start new discussion topics in theDiscussions forum.If you have time, please consider answering the polls there.

If you are working on an open-source API, you can drop us a message if you have problems using EvoMaster on it.Otherwise, if you are working in industry on closed-source APIs, we have options for academia-industry collaborations (see more info here).

• Example of generated tests
• Download and Install EvoMaster
• Build EvoMaster from source
• Command-Line Interface (CLI) options
• OpenApi/Swagger Schema
• Using EvoMaster for Black-Box Testing (easier to setup, but worse results)
• Using EvoMaster for White-Box Testing (harder to setup, but better results)
  • Write an EvoMaster Driver for White-Box Testing
  • Dealing with JDKs above version 8
• Configuring authentication
• Console output
• Library dependencies for the generated tests
• How to contribute
  • Technical notes for developers contributing to EvoMaster
• Troubleshooting
  • Windows and networking
  • java.lang.OutOfMemoryError
• More Info
  • Academic papers related to EvoMaster
  • Slides of presentations/seminars
  • Replicating studies

EvoMaster has been funded by:

• 2020-2026: a 2 million Euro grant by the European Research Council (ERC),as part of theERC Consolidator projectUsing Evolutionary Algorithms to Understand and Secure Web/Enterprise Systems.
• 2018-2021: a 7.8 million Norwegian Kroner grant by the Research Council of Norway (RCN),as part of the Frinatek projectEvolutionary Enterprise Testing.

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 864972).

EvoMaster's source code is released under the LGPL (v3) license.For a list of the used third-party libraries, you can directly see the rootpom.xml file.For a list of code directly imported (and then possibly modified/updated) fromother open-source projects, seehere.