Repository files navigation

A suite of tools including a framework for creating Java Agents, for aspect-orientedtooling for distributed systems.

Disco is a sort-of acronym ofDistributedSystemsComprehensionand may be styled as DiSCo, Disco or disco. We really don't mind.

Right now, although intended eventually to encompass other tooling in asimilar space, Disco includes a library/framework/toolkit for creatingJava Agents aimed at solving operational tooling for distributed systemsin service-oriented architectures.

Please note that whilst on versions less than semantic version 1.0, Disco ispre-release or preview software. Some APIs may be subject to change or removal untila stable v1.0 release occurs.

The Disco Java Agent toolkit provides 2 key runtime primitives, as anextension on top of Java (or other JVM languages such as Kotlin, but Javais currently the only language with robust tests).

1. An in-process Event Bus, which advertises moments in the lifetime of atransaction oractivity in service-oriented software. For example, aservice receiving a request via HTTP will begin a timeline of such eventsat that time, concluding when it has finally delivered a response to its caller.
2. A Transactionally Scoped data dictionary called the TransactionContext.ByTransactionally Scoped we mean that this data store will be createdat the beginning of the activity lifetime as described above, and surviveuntil it concludes. Extensions are added to the Java runtime such that thisdata store will be consistent during thread handoffs.

These are explored in more detail below.

In future, more languages will be supported.

Think about a straw man API on a service-oriented system. Let's say we havea service CityInfoService, with an API "getInfoForZipCode". It delegates to two downstreamservices, WeatherService and TrafficService, whose responses it aggregates into aresponse containing weather and traffic information for a given zip code. Since thesetwo services are orthogonal and not interdependent, it calls them in parallel.

The timeline of a call to getInfoForZipCode might be as follows:

1. At time T in Thread 0, a Servlet.service() call is made upon receipt of the getInfoForZipCodeAPI being called.
2. At time T+1 in Thread 0, the service submits two tasks to a pooled ExecutorService, to callthe two downstream services.
3. At time T+2 in Thread 1, a call to WeatherService is made.
4. At time T+2 in Thread 2, a call to TrafficService is made.
5. At time T+3 in Thread 1, the response from WeatherService is received.
6. At time T+4 in Thread 2, the response from TrafficService is received.
7. At time T+5 in Thread 0, Threads 1 and 2 are join()'d.
8. Finally, at time T+6 in Thread 0, a response is given to the service's caller.

The disco event bus issues events at all the key moments on this timeline of events.At the time of each event, the TransactionContext (see below) is available to passdata between the points at which each event is received and throughout this timelineof events, the Transaction ID (again, see below) is consistent and unique. If two callsto getInfoForZipCode were happening concurrently, and each had the same sequence of eventsthose events would agree on the content of the TransactionContextwithin each activitywith no crosstalk or confusion across the two.

Events are published for the upstream request and response, the downstream requests and responsesand each time a Thread is forked or joined.

Consider when a service activity might call 3downstream services,parallelized in 3 threads from a thread pool. It may do this, for example,by dispatching work to an ExecutorService.

One problem we have observed in tooling such as AWS X-Ray, due to deficienciesin Java, is that these worker threads have no clear concept of 'caller' or 'parent'.Using Disco, the Java runtime is extended to ensure that the concept of caller/parentis passed from thread to thread at the time of thread handoff (e.g. when calling Thread.start(),or someExecutor.submit(), or when using a Java 8 parallel stream), by the 'forking' threadgiving the 'forked' thread access to its Transaction Context data store.

By default, upon creation, the Transaction Context always contains a 96 bit random numberformatted as a hexadecimal string as a Transaction ID. This can be overridden by plugins orclient code if desirable, and any other arbitrary data may also be added atany point in the lifetime of the service activity. Once the data is placedin the Transaction Context, it becomes available across the activity's familyof threads thereafter.

Let's start with a simple example of a logger. In our CityInfoService above, it may bechallenging to produce really good 'joined-up' logs due to the concurrent behavior of the service.When logging the calls to WeatherService and TrafficService, the threads appear'orphaned'. If you've ever tried to make sense of logs by looking at 'nearby timestamps'instead of having unambiguous IDs available, this will be a familiar problem.

So the simplest example, is to create a Listener on the disco Event Bus which logsevery event, along with the ID from the Transaction Context. Now without taking anyparticular special action in the service's business logic itself, all these lines of logcan be tied together via this ID.

Another common requirement is to pass metadata (perhaps via HTTP headers) from service toservice, when creating tracing our routing tools in service-oriented architectures.Incoming request events and outgoing request events provide a facility to inspect, andmanipulate HTTP headers. The AWS X-Ray agent uses this feature to propagate its segmentsacross service call hops, without user-intervention.

There are two basic ways to create a Java Agent using Disco. As a self-containedartifact, or using a plugin-based system to allow multi-tenancy.

See the subproject disco-java-agent-example as a simple example of buildinga self contained agent, along with the associated tests for it in disco-java-agent-example-test

Alternatively, the plugin-based approach may be seen in the disco-java-agent-web-pluginproject, which uses the canonical agent found in the disco-java-agent/disco-java-agent packageas a substrate for plugin discovery.

As can be see in the build.gradle.kts files of several subprojects (e.g. the tests fordisco-java-agent-example), a single argument needs to be passed to the Java processat start-up. See AgentConfigParser.java in the Core libraries for details on the command linearguments for agents, and see PluginDiscovery.java for details on how a substrate agentmay load plugins.

One complexity for some managed runtimes is that the user does not have completeauthority over the arguments passed to Java. To remedy this, it is possible to 'Inject'a Disco agent at runtime, using a tiny (1 or 2 lines) amount of boilerplate code.An example of this is given in the disco-java-agent-example-inject-test project.If using this method, care must be taken to perform the injection as early as possiblein the target software's lifetime (first line of Main() is ideal, as soon as possibleafter that is the next best). Disco works on the basis of extending the Java runtimevia aspect-oriented bytecode manipulation, and some of these treatments are unable to workif the instrumented class has already been used.

This library is licensed under the Apache 2.0 Software License.

The simplest way to start is to run the following command in the root directory../gradlew build

This will build all the code, and run all the tests (functional tests and integration tests).

The final tests which are executed are tests for the 'thread handoff' TransactionContextpropogation mentioned elsewhere in this README, which deserve a mention. Some of the tests are naturally'flaky'. This is true because when we request, for example, someCollection.parallelStream(), and then performwork, the test is not in control ofwhether that will actually be parallel or actually be serial. TheJava runtime is in charge and is not easily manipulated. If the Java runtime elects not to parallelize,our test becomes meaningless - we cannot assert that disco's thread hand-off behavior iscorrect if no thread hand-off occurred at all. So these tests are designed to retry. To be clear they don'tstubbornly "retry until they succeed". They retry untilpreconditions are met which they do not control.

Unfortunately this can mean that theysometimes fail and require restarting. We don't know a better way, sorry.

Disco is available in Maven Central. A Bill of Materials (BOM) package is vended to make depending on multipleDisco packages easier.

<dependencyManagement>    <dependencies>        <dependency>            <groupId>software.amazon.disco</groupId>            <artifactId>disco-toolkit-bom</artifactId>            <version>0.13.0</version>            <type>pom</type>            <scope>import</scope>        </dependency>    </dependencies></dependencyManagement><dependencies>    <dependency>      <groupId>software.amazon.disco</groupId>      <artifactId>disco-java-agent-api</artifactId>    </dependency><!-- Other Disco dependencies--></dependencies>

implementation platform('software.amazon.disco:disco-toolkit-bom:0.13.0')implementation'software.amazon.disco:disco-java-agent-api'// Other disco dependencies

implementation(platform("software.amazon.disco:disco-toolkit-bom:0.13.0"))implementation("software.amazon.disco:disco-java-agent-api")// Other disco dependencies

If you receive the following error

Could not determine the dependencies of task ':disco-java-agent-example:shadowJar'.> Could not resolve all files for configuration ':disco-java-agent-example:runtimeClasspath'.   > path may not be null or empty string. path='null'

Please ensure that the default Java binary that is ran is Java 8. As a workaround, you may specify theJAVA_HOME to point to another version of Java.

Example:export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_202.jdk/Contents/Home

Disco is quite componentized, added to which there are quite a few projects which serve as examples and tests.

Browse through the READMEs included with the subprojects to make sense of it all, but in summary there area few layers and families of projects in here:

1. Public API contained in disco-java-agent-api
2. Special API for implementors of interception plugins, in disco-java-agent-plugin-api
3. Core library, for consumption by plugin authors and agent authors (not client code) in disco-java-agent-core
4. Canonical Pluggable agent, capable of plugin discovery, in disco-java-agent/disco-java-agent
5. A facility to 'Inject' a Disco Agent into managed runtimes like AWS Lambda
6. A Plugin to support Servlets and Apache HTTP clients, in disco-java-agent-web-plugin
7. A Plugin to support SQL connections & queries using JDBC, in disco-java-agent-sql-plugin
8. A Plugin to support requests made with the AWS SDK for Java, in disco-java-agent-aws-plugin
9. Example code in anything with '-example' in the project name.
10. Tests in anything with '-test' in the project name.

Subprojects themselves also encapsulate their own tests.