Repository files navigation

Prometheus Monitoring for Java Web Applications without Modifying their Source Code.

Thepromagent-maven-plugin is a tool for creatingcustom Java agents forPrometheus monitoring.The Java agents instrument Java Web Applications withPrometheus metrics without modifying the applications' source code.The agents use theByte Buddy bytecode manipulation library to insert Prometheus metrics during application startup.

The Promagent code repository contains two projects:

• promagent-framework: Provides, among other things, thepromagent-api and thepromagent-maven-plugin helping you to create your own agents.
• promagent-example: An example agent providing metrics forSpring Boot andJava EE applications:
  • HTTP: Number and duration of web requests.
  • SQL: Number and duration of database queries (including the HTTP context if the query was triggered by a REST call).

The example agent was tested withTomcat for theSpring Boot example and with theWildfly application server for theJava EE example.

ClonePromagent from GitHub:

git clone https://github.com/fstab/promagent.gitcd promagent

Thepromagent-api andpromagent-maven-plugin are not on Maven Central yet. Run the following commands to make them available locally (in~/.m2/repository/):

cd promagent-frameworkmvn clean installcd ..

Compile the example agent. This should create the file./promagent-example/target/promagent.jar:

cd promagent-examplemvn clean verifycd ..

The following runs with Java 8 and was not tested with Java 9 yet.

Download and compile aSpring Boot Getting Started application.

git clone https://github.com/spring-guides/gs-accessing-data-rest.gitcd gs-accessing-data-rest/completemvn clean packagecd ../..

Run the Spring Boot application with the Promagent attached.

java \    -javaagent:promagent/promagent-example/target/promagent.jar=port=9300 \    -jar gs-accessing-data-rest/complete/target/gs-accessing-data-rest-0.1.0.jar

Go tohttp://localhost:8080 to view the Spring Boot application,go tohttp://localhost:9300/metrics to view the Prometheus metrics.

This demo runs with Java 8. For a Java 9 version, seeJAVA_9_DEMO.md.

Download and compile aWildfly Quickstart application.

git clone https://github.com/wildfly/quickstart.gitcd quickstart/kitchensinkmvn clean packagecd ../..

Download and extract theWildfly application server.

curl -O http://download.jboss.org/wildfly/10.1.0.Final/wildfly-10.1.0.Final.tar.gztar xfz wildfly-10.1.0.Final.tar.gz

Run the Wildfly application server with the Promagent attached.

cd wildfly-10.1.0.FinalLOGMANAGER_JAR=$(find$(pwd) -name'jboss-logmanager-*.jar')export JAVA_OPTS="    -Xbootclasspath/p:${LOGMANAGER_JAR}    -Djboss.modules.system.pkgs=org.jboss.logmanager,io.promagent.agent    -Djava.util.logging.manager=org.jboss.logmanager.LogManager    -javaagent:../promagent/promagent-example/target/promagent.jar=port=9300"./bin/standalone.sh

In a new Shell window, deploy the quickstart application.

cd wildfly-10.1.0.Final./bin/jboss-cli.sh --connect --command="deploy ../quickstart/kitchensink/target/kitchensink.war"

Go tohttp://localhost:8080/kitchensink to view the quickstart application,go tohttp://localhost:9300/metrics to view the Prometheus metrics.

A Promagent is implemented as a set of Hooks. A Hook is a Java class meeting the following requirements:

• The class is annotated with@Hook.
• The class has a public constructor taking a single parameter of typeMetricsStore.
• The class provides methods annotated with@Before or@After. Those methods must take exactly the same parameters as the method you want to intercept (there is one exception to that rule:@Afer methods may have two additional parameters annotated with@Returned and@Thrown, seeHook Annotations below).

The best way to get started is to have a look at theServletHook andJdbcHook in thepromagent-example.

A simple Hook counting the number of Servlet requests looks as follows:

@Hook(instruments ="javax.servlet.Servlet")publicclassServletHook {privatefinalCounterservletRequestsTotal;publicServletHook(MetricsStoremetricsStore) {servletRequestsTotal =metricsStore.createOrGet(newMetricDef<>("servlet_requests_total",                    (name,registry) ->Counter.build()                        .name(name)                        .help("Total number of Servlet requests.")                        .register(registry)            ));        }@After(method ="service")publicvoidafter(ServletRequestrequest,ServletResponseresponse) {httpRequestsTotal.inc();        }}

To build a Promagent project with Maven, you need two entries in thepom.xml. First, thepromagent-api must be included as a dependency:

<dependency>    <groupId>io.promagent</groupId>    <artifactId>promagent-api</artifactId>    <version>1.0-SNAPSHOT</version>    <scope>provided</scope><!-- provided at runtime by the internal agent implementation--></dependency>

Second, thepromagent-maven-plugin that creates an agent JAR:

<build>    <finalName>promagent</finalName>    <plugins>        <plugin>            <groupId>io.promagent</groupId>            <artifactId>promagent-maven-plugin</artifactId>            <version>1.0-SNAPSHOT</version>            <executions>                <execution>                    <id>promagent</id>                    <phase>package</phase>                    <goals>                        <goal>build</goal>                    </goals>                </execution>            </executions>        </plugin>    </plugins></build>

With these two things included,mvn clean package should produce a working Java agent intarget/promagent.jar.

A Hook's lifecycle depends on whether the instrumented method call is anested call or not. A call isnested when the method is called by another method that was instrumented with the same hook.This happens for example if a Servlet's service() method calls another Servlet's service() method.

By default,nested calls are ignored and the Hook is only invoked for the outer call.In the Servlet example, this is the intended behavior, because it guarantees that each HTTP request is counted only once, even even if a Servlet internally calls another Servlet to handle the request.

If the Hook is defined with@Hook(skipNestedCalls = false) the Hook will be invoked for allnested calls, not only for the outer call.

For each outer call, a new Hook instance is created. If the Hook implements both a@Before and an@After method, the same instance is used for@Before and@After. That way, you can set a start time as a member variable in the@Before method, and use it in the@After method to calculate the duration of the call.

Fornested calls, the Hook instance from the outer call is re-used. That way, you can put data into member variables in order to pass that data down the call stack.

Most applications use static variables to maintain Prometheus metrics, as described in thePrometheus Client Library for Java documentation:

#Doesn't work with PromagentstaticfinalCountercounter =Counter.build()    .name("requests_total")    .help("Total requests.")    .register();

Unfortunately, static variables are maintained per deployment in an application server. When an application is re-deployed, a new instance of the sameCounter is created, which causes conflicts in the Prometheus registry (as the Prometheus registry is maintained by Promagent, it survives re-deployments).Moreover, it is impossible to instrument a mix of internal modules (like an internal Servlet in the JAX-RS implementation) and deployments (like Servlets in a WAR file) with static variables.

To prevent this, Promagent requires Hooks to use theMetricsStore to maintain metrics:

#ThisisthecorrectwaywithPromagentCountercounter =metricsStore.createOrGet(newMetricDef<>("requests_total",                    (name,registry) ->Counter.build()                        .name(name)                        .help("Total requests.")                        .register(registry)            ));

The Promagent library will take care that theCounter is created only once, and that theCounter instance is re-used across multiple deployments and internal modules in an application server.

• @Hook: Hook classes are annotated with@Hook(instruments = {...}, skipNestedCalls = true). Theinstruments parameter takes a list of Strings specifying the names of the classes or interfaces to be instrumented, like{"javax.servlet.Servlet", "javax.servlet.Filter"}. The Hook instruments not only the classes or interfaces themselves, but all sub-classes or implementations of these classes or interfaces. TheskipNestedCalls parameter is described inA Hook's Life Cycle above.
• @Before: Hook methods annotated with@Before(method = {...}) are invoked when an instrumented method is entered. Themethod parameter takes a list of Strings specifying the names of the intercepted methods, like{"service", "doFilter"}. The number and types of arguments are derived from the method itself, i.e. the Hook method annotated with@Before must take the exact same parameters as the methods it wants to instrument.
• @After: Hook methods annotated with@After(method = {...}) are invoked when an instrumented method is left.@After methods are always called, even if the instrumented method terminates with an Exception. The semantics is the same as with the@Before annotation. Methods annotated with@After may have two additional parameters, one parameter annotated with@Returned and one parameter annotated with@Thrown. These parameters are ignored when determining the signature of the instrumented method.
• @Returned: It might be useful to learn the return value of an instrumented method. In order to do so, methods annotated with@After may have an additional parameter annotated with@Returned, where the type corresponds to the return type of the intercepted method. If the instrumented method returns regularly, the return value is provided. If the method returns exceptionally,null (or the default type for primitive types, like0 forint) is provided.@Returned parameters are only allowed in@After methods, not in@Before methods.
• @Thrown: The@Thrown annotation is like@Returned, but to learn an Exception thrown from an instrumented method. The type should beThrowable to avoid class cast errors on unexpected RuntimeExceptions or Errors. If the instrumented method does not throw an exception, the parameter annotated with@Thrown will benull.

The Prometheus server internally stores one time series for each observed set of label values. The time series database in the Prometheus server can easily handle thousands of different time series, but millions of different time series could be a problem.Therefore, it is important to keep the number of different label values relatively small. Unique user IDs, timestamps, or session keys should not be used as label values.

Thepromagent-example strips HTTP URLs and SQL queries to make sure that there are not too many different label values:

• For HTTP requests, path parameters are replaced with placeholders. The goal is to use REST resources like/item/{id} as labels, not in actual paths like/item/123.
• For SQL queries, values are stripped. The goal is to use the structure of the query likeINSERT INTO ITEMS (ID, NAME, PRICE) VALUES (...) as a label, not in its values likeINSERT INTO ITEMS (ID, NAME, PRICE) VALUES (23, 'abc', 17.5).

Of course, replacing path parameters and SQL values is application specific. Thepromagent-example implements a very simple replacement inServletHook.stripPathParameters() andJdbcHook.stripValues(), but you probably need to customize these methods for your application.

Thepromagent-example project contains an alternative Maven configuration inpom-with-docker-tests.xml.This configuration uses thedocker-maven-plugin to create Docker images and run integration tests against Docker containers.

The Wildfly tests can be run as follows:

cd promagent-examplemvn -f pom-with-docker-tests.xml clean verify -Pwildflycd ..

The Spring Boot tests can be run as follows:

cd promagent-examplemvn -f pom-with-docker-tests.xml clean verify -Pspringcd ..

The first run takes a while, because the Docker images need to be built. Once the images are available on the local systems, runs are significantly faster.

Promagent supports three different ways of exposing metrics to the Prometheus server:

• The agent has a built-in HTTP server. This is used in the examples above. The server is started when thecommand line argumentport is used, as for example-javaagent:agent.jar=host=localhost,port=9300.Thehost argument is optional, it defaults to the wildcard IP address.Ifport is omitted the built-in server is not started.
• Thepromagent-exporter module implementsa simple Web application in WAR file format. If you deploy thepromagent-framework/promagent-exporter/target/promagent.war on yourserver, it will collect Promagent metrics via JMX and expose them under its deployment URL,likehttp://localhost:8080/promagent.
• All metrics are made available via JMX, so any JMX client can be used to access the metrics.

This is a demo project. The main goal is to learn the internals of bytecode manipulation and class loading in Java application servers. I am planning to work on the following:

• Try it with more application servers (Payara,TomEE) and adjust the code if necessary.
• Write documentation about the internal implementation, mainly the bytecode manipulation and class loading aspects.
• Generalize the concept so that users can not only write Hooks, but also other collectors. A proof-of-concept is includes in the classJmxCollector inpromagent-example.
• Generalize the concept so we don't only support monitoring with Prometheus, but also tracing with OpenTracing API compatible tools.

Thepromagent-api andpromagent-maven-plugin are not yet available on Maven Central, but they will be uploaded when the API becomes a bit more stable.

If you want to write your own agent and are looking for examples of methods you might want to instrument, look at related projects, likeinspectIT (hooks are configuredhere) orstagemonitor.

• Prometheus Monitoring for Java Web Applications w o Modifying Source Code (youtube.com), video from my Devoxx 2017 talk.
• Instrumenting Java Web Applications without Modifying their Source Code (Java Code Geeks).

This project is supported as part of the R&D activities atConSol Software GmbH. See theConSol Labs Blog for more info.