Posted onNov 13, 2022 • Edited onNov 15, 2022

Spring Boot – Black Box Testing

#java #spring #testing #tutorial

In this article, I’m showing you

What’s the difference between white box and black testing
What are the benefits of the latter
How you can implement it in your Spring Boot application
How to configure the OpenAPI generator to simplify code and reduce duplications

You can find the code examples inthis repository.

Domain

We’re developing a restaurant automatization system. There are two domain classes.Fridge andProduct. A fridge can have many products, whilst a product belongs to a single fridge. Look at the classes declaration below.

@Entity@Table(name="fridge")publicclassFridge{@Id@GeneratedValue(strategy=IDENTITY)privateLongid;privateStringname;@OneToMany(fetch=LAZY,mappedBy="fridge")privateList<Product>products=newArrayList<>();}@Entity@Table(name="product")publicclassProduct{@Id@GeneratedValue(strategy=IDENTITY)privateLongid;@Enumerated(STRING)privateTypetype;privateintquantity;@ManyToOne(fetch=LAZY)@JoinColumn(name="fridge_id")privateFridgefridge;publicenumType{POTATO,ONION,CARROT}}

I'm usingSpring Data JPA as a persistence framework. Therefore, those classes areHibernate entities.

White box testing

This type of testing makes an assumption that we know some implementation details and may interact them. We have 4 REST API endpoints in the system:

Create a newFridge.
Add a newProduct.
Change theProduct quantity.
Remove theProduct from theFridge.

Suppose we want to test the one that changes theProduct quantity. Take a look at the example of the test below.

@SpringBootTest(webEnvironment=RANDOM_PORT)classProductControllerWhiteBoxTestextendsIntegrationSuite{@AutowiredprivateFridgeRepositoryfridgeRepository;@AutowiredprivateProductRepositoryproductRepository;@AutowiredprivateTestRestTemplaterest;@BeforeEachvoidbeforeEach(){productRepository.deleteAllInBatch();fridgeRepository.deleteAllInBatch();}@TestvoidshouldUpdateProductQuantity(){finalvarfridge=fridgeRepository.save(Fridge.newFridge("someFridge"));finalvarproductId=productRepository.save(Product.newProduct(POTATO,10,fridge)).getId();assertDoesNotThrow(()->rest.put("/api/product/{productId}?newQuantity={newQuantity}",null,Map.of("productId",productId,"newQuantity",20)));finalvarproduct=productRepository.findById(productId).orElseThrow();assertEquals(20,product.getQuantity());}}

Let’s examine this piece of code step by step.
We inject repositories to manipulate rows in the database. Then theTestRestTemplate comes into play. This bean is used to send HTTP requests. Then you can see that the@BeforeEach callback deletes all rows from the database. So, each test runs deterministically. And finally, here is the test itself:

We create a newFridge.
Then we create a newProduct with a quantity of10 that belongs to the newly createdFridge.
Afterwards, we invoke the REST endpoint to increase theProduct quantity from10 to20.
Eventually we select the sameProduct from the database and check that the quantity has been increased.

The test works fine. Anyway, there are nuances that should be taken into account:

Though the test verifies the entire system behavior (aka functional test) there is a coupling on implementation details (i.e. the database).
What the test validates is not the actual use case. If somebody wants to interact with our service, they won’t be able to insert and update rows in the database directly.

As a matter of fact, if we want to test the system from the user perspective, we can only use the public API that the service exposes.

Black box testing

This type of testing means loose coupling on the system's implementation details. Therefore, we can only depend on the public API (i.e. REST API).

Check out the previous white box test example. How can we refactor it into the black box kind? Look at the@BeforeEach implementation below.

@BeforeEachvoidbeforeEach(){productRepository.deleteAllInBatch();fridgeRepository.deleteAllInBatch();}

A black box test should not interact with the persistence provider directly. Meaning that there should be a separate REST endpoint clearing all data. Look at the code snippet below.

@RestController@RequiredArgsConstructor@Profile("qa")publicclassQAController{privatefinalFridgeRepositoryfridgeRepository;privatefinalProductRepositoryproductRepository;@DeleteMapping("/api/clearData")@TransactionalpublicvoidclearData(){productRepository.deleteAllInBatch();fridgeRepository.deleteAllInBatch();}}

Now we have a particular controller that encapsulates all the clearing data logic. If the test depends only on this endpoint, then we can safely put changes and refactor the method as the application grows. And our black box tests won’t break. The@Profile(“qa”) annotation is crucial. We don’t want to expose an endpoint that can delete all user data in production or even development environment. So, we register this endpoint, ifqaprofile is active. We’ll use it only in tests.

Theqa abbreviation stands for thequality assurance.

And now we should refactor the test method itself. Have a look at its implementation below again.

@TestvoidshouldUpdateProductQuantity(){finalvarfridge=fridgeRepository.save(Fridge.newFridge("someFridge"));finalvarproductId=productRepository.save(Product.newProduct(POTATO,10,fridge)).getId();assertDoesNotThrow(()->rest.put("/api/product/{productId}?newQuantity={newQuantity}",null,Map.of("productId",productId,"newQuantity",20)));finalvarproduct=productRepository.findById(productId).orElseThrow();assertEquals(20,product.getQuantity());}

There are 3 operations that should be replaced with direct REST API invocations. These are:

Creating newFridge.
Creating newProduct.
Checking the theProduct quantity has been increased.

Look at the whole black box test example below.

@SpringBootTest(webEnvironment=RANDOM_PORT)@ActiveProfiles("qa")classProductControllerBlackBoxTestextendsIntegrationSuite{@AutowiredprivateTestRestTemplaterest;@BeforeEachvoidbeforeEach(){rest.delete("/api/qa/clearData");}@TestvoidshouldUpdateProductQuantity(){// create new FridgefinalvarfridgeResponse=rest.postForEntity("/api/fridge?name={fridgeName}",null,FridgeResponse.class,Map.of("fridgeName","someFridge"));assertTrue(fridgeResponse.getStatusCode().is2xxSuccessful(),"Error during creating new Fridge: "+fridgeResponse.getStatusCode());// create new ProductfinalvarproductResponse=rest.postForEntity("/api/product/fridge/{fridgeId}",newProductCreateRequest(POTATO,10),ProductResponse.class,Map.of("fridgeId",fridgeResponse.getBody().id()));assertTrue(productResponse.getStatusCode().is2xxSuccessful(),"Error during creating new Product: "+productResponse.getStatusCode());// call the API that should be testedassertDoesNotThrow(()->rest.put("/api/product/{productId}?newQuantity={newQuantity}",null,Map.of("productId",productResponse.getBody().id(),"newQuantity",20)));// get the updated Product by idfinalvarupdatedProductResponse=rest.getForEntity("/api/product/{productId}",ProductResponse.class,Map.of("productId",productResponse.getBody().id()));assertTrue(updatedProductResponse.getStatusCode().is2xxSuccessful(),"Error during retrieving Product by id: "+updatedProductResponse.getStatusCode());// check that the quantity has been changedassertEquals(20,updatedProductResponse.getBody().quantity());}}

The benefits of black box testing in comparison to white box testing are:

The test checks the path that a user shall do to retrieve the expected result. Therefore, the verification behavior becomes more robust.
Black box tests are highly stable against refactoring. As long as the API contract remains the same, the test should not break.
If you accidentally break the backward compatibility (e.g. adding a new mandatory parameter to an existing REST endpoint), the black box test will fail and you'll determine the issue way before the artefact is being deployed to any environment.

However, there is a slight problem with the code that you have probably noticed. The test is rather cumbersome. It’s hard to read and maintain. If I didn’t put in the explanatory comments, you would probably spend too much time figuring out what’s going on. Besides, the same endpoints might be called for different scenarios, which can lead to code duplication.

Luckily there is solution.

OpenAPI and code generation

Spring Boot comes with a brilliantOpenAPI support. All you have to do is to add two dependencies. Look at theGradle configuration below.

implementation'org.springframework.boot:spring-boot-starter-actuator'implementation'org.springdoc:springdoc-openapi-ui:1.6.12'

After adding these dependencies, the OpenAPI specification is available byGET /v3/api-docs endpoint.

TheSpringDoc library comes with lots of annotations to tune your REST API specification precisely. Anyway, that's out of context of this article.

If we have the OpenAPI specification, it means that we can generate Java classes to call the endpoints in a type-safe manner. What’s even more exciting is that we can apply those generated classes in our black box tests!

Firstly, let's define the requirements for the upcoming OpenAPI Java client:

The generated classes should be put into.gitignore. Otherwise, if you haveCheckstyle,PMD, orSonarQube in your project, then generated classes can violate some rules. Besides, if you don't put them into.gitignore, then each pull request might become huge due to the fact that even a slightest fix can lead to lots of changes in the generated classes.
Each pull request build should guarantee that generated classes are always up to date with the actual OpenAPI specification.

How can we get the OpenAPI specification itself during the build phase? The easiest way is to write a separate test that creates the web part of the Spring context, invokes the/v3/api-docs endpoint, and put the retrieved specification intobuild folder (if you are Maven user, then it will betarget folder). Take a look at the code example below.

@SpringBootTest(webEnvironment=RANDOM_PORT)@AutoConfigureTestDatabase@ActiveProfiles("qa")publicclassOpenAPITest{@AutowiredprivateTestRestTemplaterest;@Test@SneakyThrowsvoidgenerateOpenApiSpec(){finalvarresponse=rest.getForEntity("/v3/api-docs",String.class);assertTrue(response.getStatusCode().is2xxSuccessful(),"Unexpected status code: "+response.getStatusCode());// the specification will be written to 'build/classes/test/open-api.json'Files.writeString(Path.of(getClass().getResource("/").getPath(),"open-api.json"),response.getBody());}}

The@AutoConfigureTestDatabase configures the in-memory database (e.g.H2), if you have one in the classpath. Since the database provider does not affect the result OpenAPI specification, we can make the test run a bit faster by not usingTestcontainers.

Now we have the result specification. How can we generate the Java classes based on it? We have another Gradle plugin for that. Take a look at thebuild.gradle configuration below.

plugins{...id"org.openapi.generator"version"6.2.0"}...openApiGenerate{inputSpec="$buildDir/classes/java/test/open-api.json".toString()outputDir="$rootDir/open-api-java-client".toString()apiPackage="com.example.demo.generated"invokerPackage="com.example.demo.generated"modelPackage="com.example.demo.generated"configOptions=[dateLibrary:"java8",openApiNullable:"false",]generatorName='java'groupId="com.example.demo"globalProperties=[modelDocs:"false"]additionalProperties=[hideGenerationTimestamp:true]}

In this article, I'm showing you how to configure the correspondingGradle plugin. Anyway, there isMaven plugin as well and the approach won't be different much.

There is an important detail about OpenAPI generator plugin. It creates the whole Gradle/Maven/SBT project (containingbuild.gradle,pom.xml, andbuild.sbt files) but not just Java classes. So, we set the theoutputDir property as$rootDir/open-api-java-client. Therefore, the generated Java classes go into the Gradle subproject.

We should also mark theopen-api-java-client directory as a subproject in thesettings.gradle. Look at the code snippet below.

rootProject.name='demo'include'open-api-java-client'

All you have to do to generate OpenAPI Java client is to run these Gradle commands:

gradletest--tests"com.example.demo.controller.OpenAPITest.generateOpenApiSpec"gradle openApiGenerate

Applying the Java client

Now let’s try our brand new Java client in action. We’ll create a separate@TestComponent for convenience. Look at the code snippet below.

@TestComponentpublicclassTestRestController{@AutowiredprivateEnvironmentenvironment;publicFridgeControllerApifridgeController(){returnnewFridgeControllerApi(newApiClient());}publicProductControllerApiproductController(){returnnewProductControllerApi(newApiClient());}publicQaControllerApiqaController(){returnnewQaControllerApi(newApiClient());}privateApiClientnewApiClient(){finalvarapiClient=newApiClient();apiClient.setBasePath("http://localhost:"+environment.getProperty("local.server.port",Integer.class));returnapiClient;}}

Finally, we can refactor our black box test. Look at the final version below.

@SpringBootTest(webEnvironment=RANDOM_PORT)@ActiveProfiles("qa")@Import(TestRestControllers.class)classProductControllerBlackBoxGeneratedClientTestextendsIntegrationSuite{@AutowiredprivateTestRestControllersrest;@BeforeEach@SneakyThrowsvoidbeforeEach(){rest.qaController().clearData();}@TestvoidshouldUpdateProductQuantity(){finalvarfridgeResponse=assertDoesNotThrow(()->rest.fridgeController().createNewFridge("someFridge"));finalvarproductResponse=assertDoesNotThrow(()->rest.productController().createNewProduct(fridgeResponse.getId(),newProductCreateRequest().quantity(10).type(POTATO)));assertDoesNotThrow(()->rest.productController().updateProductQuantity(productResponse.getId(),20));finalvarupdatedProduct=assertDoesNotThrow(()->rest.productController().getProductById(productResponse.getId()));assertEquals(20,updatedProduct.getQuantity());}}

As you can see, the test is much more declarative. Moreover, the API contract become statically typed and the parameters validation proceeds during compile time!

The`.gitignore` caveat and the separate test source

I told that we should put the generated classes to the.gitignore. However, if you mark theopen-api-java-client/src directory as the unindexed by Git, then you suddenly realize that your tests do not compile in CI environment. The reason is that the process of generation the OpenAPI specification (i.e.open-api.json file) is an individual test as well. And even if you tell the Gradle to run a single test directly, it will compile everything in thesrc/test directory. In the end, tests don’t compile successfully.

Thankfully, the issue can be solved easily. Gradle providessource sets. It's a logical group that splits the code into separate modules that you can compile independently.

Firstly, let's add thegradle-testsets plugin and define a separate test source that'll contain theOpenAPITest file. It's the one that generates theopen-api.json specification. Take a look at the code example below.

plugins{...id"org.unbroken-dome.test-sets"version"4.0.0"}...testSets{openApiGenerator}tasks.withType(Test){group='verification'useJUnitPlatform()testLogging{showExceptionstrueshowStandardStreams=falseshowCausestrueshowStackTracestrueexceptionFormat"full"events("skipped","failed","passed")}}openApiGenerator.outputs.upToDateWhen{false}tasks.named('openApiGenerate'){dependsOn'openApiGenerator'}

ThetestSets block declares a new source set calledopenApiGenerator. Meaning that Gradle treats thesrc/openApiGenerator directory like another test source.

Thetasks.withType(Test) declaration is also important. We need to tell Gradle that every task ofTest type (i.e. thetest task itself and theopenApiGenerator as well) should run with JUnit.

I put theupToDateWhen option for convenience. It means that the test that generatesopen-api.json file will be always run on demand and never cached.

And the last block defines that before generating the OpenAPI Java client we should update the specification in advance.

Now we just need to move theOpenAPITest to thesrc/openApiGenerator directory and also make a slight change to theopenApiGenerate task inbuild.gradle. Look at the code snippet below.

openApiGenerate{// 'test' directory should be replaced with 'openApiGenerator'inputSpec="$buildDir/classes/java/openApiGenerator/open-api.json".toString()....}

Finally, you can build the entire project with these two commands.

gradle openApiGenerategradle build

Conclusion

The black box testing is a crucial part of the application development process. Try it and you’ll notice that the test scenarios become much more representative. Besides, black box test are also great documentation for the API. You can even applySpring REST Docs and generate a nice manual that’ll be useful both for the API users and QA engineers.

If you have any questions or suggestion, leave your comments down below. Thanks for reading!