The Pipeline design pattern (in Java)
Recently on a project, I was looking to break down a large operation into separate pieces that could be tested and executed independently. The operation itself was complex to the point where it had become unmanageable; each successive update to the operation was adding regressions to the code. In addition, each update required the developer to write a large number of unit tests to check many permutations of inputs. This piece of the codebase was eating up the most developer time to maintain.
I tried refactoring the operation using instance methods that all lived within the operation’s class, so that the operation would be a series of successive method calls, but this didn’t quite “fit” right. Even moving each of the instance methods into it’s own helper class felt like it wasn’t the right code organization. There were many exceptions being thrown in the code (I’m a huge proponent of theClean Code byRobert C. Martin) and trying to find the called methods inside bloated try-catch handlers became difficult.
I tried to look through Design Patterns online, specifically those that were created by theGang of Four(you can search this term), and the closest thing I could find was theChain of Responsibility pattern. However, the classic version of this pattern would have required me to mutate the input at each step. What I wanted was something moreFunctional, where inputs were immutable and each step would output a result to be used by the next step (similar to Java streams for collections). Thankfully I managed to stumble uponthis stackoverflow post, that brought me a solution.
So here is the Pipeline Design Pattern in Java, pulling from elements of functional programming and making use of Java 8 lambdas.
The interface that each “step” of the Pipeline implements:
This is followed by the actual “pipeline” which runs each step in series, using the output of one for the input of the next step:
Let’s now take an example. Suppose we wanted to take two integers, add them together, and return the string representation of the result. The code might look as follows:
There are some caveats to thePipeline design pattern.
- The pattern can lead to over architecting the inputs and output. It is a ton of work in java to create the inputs and outputs needed for a very complex pipeline. However, I feel that creating the inputs and outputs makes you think about what each step is ingesting. I’ve found that my steps are broken into better, more modular (and testable) pieces versus an more iterative approach.
- There will be initial input values that need to be passed to each output for each step because they are not used until later in the pipeline. This is ok, although for some people it may feel more disorganized.
In my humble opinion, thePipeline design patternis a useful pattern for organizing complex operations in code where initial values get converted into intermediate values that get turned into final values. It takes really nice aspects of functional programming and brings it to more iterative OOP practices. I will definitely be using this pattern more often now that I’ve seen it work.
