Writing integration tests is not always straightforward. By definition, integration tests require interaction between several components, and we need to deal with them in many different ways. Let’s look at some tools that will make writing and reading tests easier. I believe thatTestcontainers and theSpark Framework will allow you to write shorter and more descriptive tests.

Is this how you test?

What is your approach to writing an integration test (IT)? Maybe some of the following sound familiar:

• Write mocks or stubs of external services
• Create a dedicated remote environment for ITs (playground, sandbox) and run them there
• Setup all the components (where the ITs are supposed to run) locally

No, I’m not saying that you’ve been doing it all wrong if you do that! But the truth is each of those approaches has drawbacks. By way of example, let’s look at the first option.

When you mock or stub some external services that are not crucial for the component you are testing, there is a chance you might miss some aspects of that mocked service that can only occur when running live.

Of course, you could invest more effort into replicating the logic of how the actual component works, but it might be difficult to replicate accurately and will also be time-consuming to develop. Even then, there are no guarantees it will be correct, so your test might still be unreliable.

What if there was a more effective way? Let’s see what we can do to makereal integration tests and not imitation ones!

Meet Testcontainers

Testcontainers is a Java library that supportsJUnit tests providing lightweight instances of anything that we can run in aDocker container.

I will go through a use case where Testcontainers can provide substantial benefits.

In the project I’m currently working on, we have a component calledthe Integration Component (IC). IC is aSpring Boot service that acts as a consumer and a producer ofRabbitMQ messages. As a consumer, it listens to a queue where another service sends job requests. IC reads those messages (job requests), processes them and finally sends an HTTP request toDatabricks to run a job. Before we submit the request (step #4 on the diagram below), we need to do a few other things, and we have divided this logic into several steps in the IC.

For testing purposes, those requests are handled by the Spark framework, but I’ll get back to that later.

As mentioned before, the service logic is divided into several steps, where each step has aprocess() method. Let’s look at theSendRequestToJobQueueStep method (#3 on the diagram above).

@Slf4j@RequiredArgsConstructor@ComponentpublicclassSendRequestToJobQueueStepimplementsJobRequestConsumerStep{privatefinalAmqpAdminamqpAdmin;privatefinalExchangeexchange;privatefinalRequestQueueProviderrequestQueueProvider;privatefinalRabbitTemplaterabbitTemplate;@Value("${config.request-queue-ttl}")privateDurationrequestQueueTtl;@Overridepublicbooleanprocess(JobRequestProcessingContextcontext){StringqueueName=createAndBindRequestQueue(runSettings.getTraceId(),context.getJobType());try{Supplier<?extendsSpecificRecordBase>requestProvider=context.getRequestProvider();sendJobRequestToRequestQueue(requestProvider.get(),queueName);}catch(AmqpExceptione){StringcustomMsg=String.format("Sending '%s' request using routing key '%s' for jobId=%d failed.",context.getJobType(),queueName,job.getJobId());log.error(prepareExceptionLogMessage(deliveryTag,e,customMsg),e);requeue(context,deliveryTag);returnfalse;}returntrue;}private<RextendsSpecificRecordBase>voidsendJobRequestToRequestQueue(RrequestObject,StringroutingKey){rabbitTemplate.convertAndSend(Amqp.EVENTS,routingKey,requestObject);log.info("Job request is sent to job queue. Routing key: '{}'",routingKey);}privateStringcreateAndBindRequestQueue(StringtraceId,JobTypejobType){QueuerequestQueue=requestQueueProvider.getRequestQueue(jobType,traceId);amqpAdmin.declareQueue(requestQueue);StringroutingKey=requestQueue.getName();Bindingbinding=BindingBuilder.bind(requestQueue).to(exchange).with(routingKey).and(Map.of("x-expires",requestQueueTtl.toMillis()));amqpAdmin.declareBinding(binding);returnroutingKey;}}

When theprocess() method is invoked, the IC is sending a job request to the dynamically created and bound queue. The creation and binding happen in thecreateAndBindRequestQueue() method.

There’s quite a lot going on in that class. Imagine writing an integration test that would cover all that logic!

There’s another challenge. Consider thecreateAndBindRequestQueue() method. If you mock all the methods used in it, namelydeclareQueue() anddeclareBinding(), will it really help you? Sure, you can verify if those methods were invoked, or try to return a value (if it’s possible), but it’s not actually the same as running the code live.

An approach using mocks might look like this:

@TestvoidqueueShouldBeDeclaredAndBoundDuringCreation(){when(queue.getName()).thenReturn(QUEUE\_NAME);when(exchange.getName()).thenReturn(EXCHANGE\_NAME);step().process(context);verify(amqpAdmin).declareQueue(queue);ArgumentCaptor<Binding>bindingArgumentCaptor=ArgumentCaptor.forClass(Binding.class);verify(amqpAdmin).declareBinding(bindingArgumentCaptor.capture());Bindingbinding=bindingArgumentCaptor.getValue();assertEquals(EXCHANGE\_NAME,binding.getExchange());assertEquals(QUEUE\_NAME,binding.getRoutingKey());assertEquals(QUEUE\_NAME,binding.getDestination());}

This might be considered a unit test, but it’s definitely not an integration test. What we need here is to verify whether the queue has been created for real and a message has been sent to it.

Here’s how to achieve that using Testcontainers.

@Slf4j@SpringBootTest@TestcontainersclassCommonJobRequestConsumerIT{privatestaticfinalintSPARK\_SERVER\_PORT=4578;@ContainerprivatestaticfinalRabbitMQContainercontainer=newRabbitMQContainer(DockerImageName.parse("rabbitmq:3.8.14-management")){@Overridepublicvoidstop(){log.info("Allow Spring Boot to finalize things (Failed to check/redeclare auto-delete queue(s).)");}};@AutowiredprivateRabbitTemplatetemplate;@Value("${databricks.check-status-call-delay}")privateDurationstatusCallDelay;privatestaticSparkServicesparkServer;@BeforeAllstaticvoidbefore(){sparkServer=SparkService.instance(SPARK\_SERVER\_PORT);sparkServer.startWithDefaultRoutes();}@AfterAllstaticvoidafter(){sparkServer.stop();}private<RextendsSpecificRecordBase>voidassertJobRequest(StringexpectedQueueName,Supplier<R>requestSupplier,JobSettingsProvider<R>jobSettingsProvider)throwsIOException{MessagereceivedRequest=template.receive(expectedQueueName);assertNotNull(receivedRequest);RserializedRequest=AvroSerialization.fromJson(requestSupplier.get(),receivedRequest.getBody());log.info("Request received in '{}' '{}'",expectedQueueName,serializedRequest.toString());RunSettingsjobSettings=jobSettingsProvider.getJobSettings(serializedRequest);assertEquals(JOB\_ID,jobSettings.getJobId());assertEquals(TRACE\_ID,jobSettings.getTraceId());}@ParameterizedTest@MethodSource<RextendsSpecificRecordBase>voidjobRequestShouldBeSentToDedicatedQueue(StringrequestRoutingKey,Rrequest,StringexpectedQueueName,Supplier<R>requestSupplier,JobSettingsProvider<R>jobSettingsProvider)throwsInterruptedException,IOException{template.convertAndSend(Amqp.EVENTS,requestRoutingKey,request,m->{m.getMessageProperties().setDeliveryTag(1);returnm;});// finish all steps// + give rabbit some time to finish with dynamic queue creationlongtimeout=statusCallDelay.getSeconds()+1;TimeUnit.SECONDS.sleep(timeout);assertJobRequest(expectedQueueName,requestSupplier,jobSettingsProvider);}}

As you can see, the test is readable and really easy to follow, which is not always the case when you mock. We start (in thebefore() method) with some Spark related logic (more on that later), and then we send a message to the queue, starting the entire process. This is exactly how the system under test (IC) works. It’s listening to a particular queue and once the message is there, it picks it up and starts processing it. In some cases, we need to wait a bit, since otherwise a test will finish too early and assertions will fail.

I think that proves that there is a good reason why using Testcontainers in similar cases could be an excellent choice. In my opinion, there is no better way to be certain this code works as expected.

I’m sure there are many other examples where mocking is not a viable solution, and the only reasonable option is to be able to run those components live. This is where the Testcontainers library shows its power and simplicity. Give it a try next time you write an integration test!

Hero #2: The Spark framework

In the same component (IC) I’m also using theSpark framework to handle HTTP calls to an external service, in this case,Databricks API. Spark is lightweight and perfect for such use cases. Instead of mockingRestTemplate calls, we are using a real HTTP server!

Why is that so important? Well, if you look at how our test is organised, I think it will become apparent. As mentioned earlier, I’m using Testcontainers to make the test as real as possible. I do not want to mock anything. I want my REST calls to be real as well.

Let’s look at how Spark is being used in this test. While working with Spark in integration tests, I’m using a wrapper class called SparkService.

@Slf4jpublicclassSparkService{privatefinalServicesparkService;privateSparkService(Serviceservice){this.sparkService=service;}staticFileloadJsonPayload(StringpayloadFileName){ClassLoaderclassLoader=SparkService.class.getClassLoader();URLresource=classLoader.getResource(payloadFileName);returnnewFile(resource.getFile());}publicstaticSparkServiceinstance(intdbServerPort){returnnewSparkService(Service.ignite().port(dbServerPort));}publicvoidstartWithDefaultRoutes(){DatabricksRoutes.JOB\_LIST.register(sparkService);DatabricksRoutes.JOB\_RUN\_NOW.register(sparkService);DatabricksRoutes.JOB\_RUNS\_GET.register(sparkService);DatabricksRoutes.JOB\_RUNS\_DELETE.register(sparkService);}publicvoidstop(){service().stop();}publicvoidawaitInitialization(){service().awaitInitialization();}publicServiceservice(){returnsparkService;}}

Notice thestartWithDefaultRoutes() method. It contains several lines where particular endpoints (which I would like to stub) are defined. I’m using enum classes for those endpoints, and each of the enum keys implements the SparkRoute interface.

publicinterfaceSparkRoute{HttpMethodhttpMethod();Stringpath();Routeroute();defaultvoidregister(ServicesparkService){register(sparkService,route());}defaultvoidregister(ServicesparkService,Routeroute){switch(httpMethod()){caseGET:sparkService.get(path(),route);break;casePOST:sparkService.post(path(),route);break;}}}

Here is an example of theJOB_LIST enum from theDatabricksRoutes class.

publicenumDatabricksRoutesimplementsSparkRoute{JOB\_LIST{@OverridepublicStringpath(){return"/api/2.0/jobs/list";}@OverridepublicRouteroute(){returnJobController.handleJobList("json/spark/jobs\_list\_response.json");}@OverridepublicHttpMethodhttpMethod(){returnHttpMethod.GET;}}}

Ok, so how are all of these used in actual integration tests? In a simple scenario where no special logic for the stubbed endpoints is required, it could look like this.

privatestaticSparkServicesparkServer;@BeforeAllstaticvoidbefore(){sparkServer=SparkService.instance(SPARK\_SERVER\_PORT);sparkServer.startWithDefaultRoutes();}@AfterAllstaticvoidafter(){sparkServer.stop();}@DynamicPropertySourcestaticvoidproperties(DynamicPropertyRegistryregistry){registry.add("databricks.url=",()->"http://localhost:"+SPARK\_SERVER\_PORT);registry.add("databricks.token.token-host=",()->"http://localhost:"+SPARK\_SERVER\_PORT);registry.add("spring.rabbitmq.host",container::getContainerIpAddress);registry.add("spring.rabbitmq.port",()->container.getMappedPort(5672));log.info("RabbitMQ console available at: {}",container.getHttpUrl());}

I have not mentioned this earlier but in theproperties() method above, you can see how RabbitMQ can be configured with Testcontainers. This method is where all the properties in which we need to specify a URL are overridden, so we could use Spark as a handler for the original REST calls to those default services.

That was a simple Spark usage scenario within an integration test. For more sophisticated logic, we need a bit of a different approach.

What if we have a parameterised test, and we need each given endpoint to return a different response for every run? In the example above, all the endpoints were defined before the test started. However, we can configure particular endpoint handlers inside each test. This is where Spark can show its power of configuration and customisation in handling incoming requests.

For instance, consider this example:

sparkServer.registerGetRoute(JOB\_RUNS\_GET\_PATH,handleRunsGet(jobRunDataPath));sparkServer.awaitInitialization();

jobRunDataPath is one of the parameters in a parameterised test, so we can register a different request handler and return a custom response (a JSON file) for every test run.

Try it out!

To sum up, I believe thatTestcontainers and theSpark framework will change your habits when writing integration tests.

By leveraging the power of containerisation, you can move your tests to the next level by making them more reliable and even easier to write. You will be able to verify your system under test in almost the same conditions as if it was running on production. Furthermore, your test can eventually become even more readable.

Give it a try and see how easy it is to write integration tests now!