Repository files navigation

Clue provides a set of Go packages for instrumenting microservices. Theemphasis is on simplicity and ease of use. Although not a requirement, Clueworks best when used in microservices written usingGoa.

Clue covers the following topics:

• Instrumentation: theclue package configures OpenTelemetryfor service instrumentation.
• Logging: thelog package provides a context-based logging API thatintelligently selects what to log.
• Health checks: thehealth package provides a simple way forservices to expose a health check endpoint.
• Dependency mocks: themock package provides a way to mockdownstream dependencies for testing.
• Debugging: thedebug package makes it possible to troubleshootand profile services at runtime.
• Interceptors: theinterceptors package provides a set ofhelpful Goa interceptors.

Clue's goal is to provide all the ancillary functionality required to efficientlyoperate a microservice style architecture including instrumentation, logging,debugging and health checks. Clue is not a framework and does not providefunctionality that is already available in the standard library or in otherpackages. For example, Clue does not provide a HTTP router or a HTTP serverimplementation. Instead, Clue provides a way to instrument existing HTTP or gRPCservers and clients using the standard library and the OpenTelemetry API.

Learn more about Clue's observability features in theObservabilitysection of thegoa.design documentation. The guide covers how to effectively monitor,debug and operate microservices using Clue's instrumentation capabilities.

• Theclue package provides a simple API for configuring OpenTelemetryinstrumentation. The package also provides a way to configure the logpackage to automatically annotate log messages with trace and span IDs.The package also implements a dynamic trace sampler that can be used tosample traces based on a target maximum number of traces per second.

• Thelog package offers a streamlined, context-based structured logger thatefficiently buffers log messages. It smartly determines the optimal time toflush these messages to the underlying logger. In its default configuration,the log package flushes logs upon the logging of an error or when a request istraced. This design choice minimizes logging overhead for untraced requests,ensuring efficient logging operations.

• Thehealth package provides a simple way to expose a health check endpointthat can be used by orchestration systems such as Kubernetes to determinewhether a service is ready to receive traffic. The package implements theconcept of checkers that can be used to implement custom health checks witha default implementation that relies on the ability to ping downstreamdependencies.

• Themock package provides a way to mock downstream dependencies for testing.The package provides a tool that generates mock implementations of interfacesand a way to configure the generated mocks to validate incoming payloads andreturn canned responses.

• Thedebug package provides a way to dynamically control the log level of arunning service. The package also provides a way to expose the pprof Goprofiling endpoints and a way to expose the log level control endpoint.

Clue requires Go 1.21 or later. Install the packages required for yourapplication usinggo get, for example:

go get goa.design/clue/cluego get goa.design/clue/loggo get goa.design/clue/healthgo get goa.design/clue/mockgo get goa.design/clue/debug

The following snippet illustrates how to useclue to instrument a HTTP server:

ctx:=log.Context(context.Background(),// Create a clue logger context.log.WithFunc(log.Span))// Log trace and span IDs.metricExporter:=stdoutmetric.New()// Create metric and span exporters.spanExporter:=stdouttrace.New()cfg:=clue.NewConfig(ctx,"service","1.0.0",metricExporter,spanExporter)clue.ConfigureOpenTelemetry(ctx,cfg)// Configure OpenTelemetry.handler:=http.HandlerFunc(func(w http.ResponseWriter,r*http.Request) {w.Write([]byte("Hello, World!"))})// Create HTTP handler.handler=otelhttp.NewHandler(handler,"service" )// Instrument handler.handler=debug.HTTP()(handler)// Setup debug log level control.handler=log.HTTP(ctx)(handler)// Add logger to request context and log requests.mux:=http.NewServeMux()// Create HTTP mux.mux.HandleFunc("/",handler)// Mount handler.debug.MountDebugLogEnabler(mux)// Mount debug log level control handler.debug.MountDebugPprof(mux)// Mount pprof handlers.mux.HandleFunc("/health",health.NewHandler(health.NewChecker()))// Mount health check handler.http.ListenAndServe(":8080",mux)// Start HTTP server.

Similarly, the following snippet illustrates how to instrument a gRPC server:

ctx:=log.Context(context.Background(),// Create a clue logger context.log.WithFunc(log.Span))// Log trace and span IDs.metricExporter:=stdoutmetric.New()spanExporter:=stdouttrace.New()cfg:=clue.NewConfig(ctx,"service","1.0.0",metricExporter,spanExporter)clue.ConfigureOpenTelemetry(ctx,cfg)// Configure OpenTelemetry.svr:=grpc.NewServer(grpc.ChainUnaryInterceptor(log.UnaryServerInterceptor(ctx),// Add logger to request context and log requests.debug.UnaryServerInterceptor()),// Enable debug log level controlgrpc.StatsHandler(otelgrpc.NewServerHandler()),// Instrument server.)

Note that in the case of gRPC, a separate HTTP server is required to expose thedebug log level control, pprof and health check endpoints:

mux:=http.NewServeMux()// Create HTTP mux.debug.MountDebugLogEnabler(mux)// Mount debug log level control handler.debug.MountDebugPprof(mux)// Mount pprof handlers.mux.HandleFunc("/health",health.NewHandler(health.NewChecker()))// Mount health check handler.gohttp.ListenAndServe(":8081",mux)// Start HTTP server.

Exporter are responsible for exporting telemetry data to a backend. TheOpenTelemetry Exporters documentationprovides a list of available exporters.

For example, configuring anOTLPcompliant span exporter can be done as follows:

import"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"// ...spanExporter,err:=otlptracegrpc.New(context.Background(),otlptracegrpc.WithEndpoint("localhost:4317"),otlptracegrpc.WithTLSCredentials(insecure.NewCredentials()))

While configuring an OTLP compliant metric exporters can be done as follows:

import"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"// ...metricExporter,err:=otlpmetricgrpc.New(context.Background(),otlpmetricgrpc.WithEndpoint("localhost:4317"),otlpmetricgrpc.WithTLSCredentials(insecure.NewCredentials()))

These exporters can then be used to configure Clue:

// Configure OpenTelemetry.cfg:=clue.NewConfig(ctx,"service","1.0.0",metricExporter,spanExporter)clue.ConfigureOpenTelemetry(ctx,cfg)

HTTP clients can be instrumented using the Cluelog and OpenTelemetryotelhttptrace packages. Thelog.Client function wraps a HTTP transport and logs the request and response. Theotelhttptrace.Client function wraps a HTTP transport and adds OpenTelemetry tracing to the request.

import ("context""net/http""net/http/httptrace""go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp""go.opentelemetry.io/contrib/instrumentation/net/http/otelhttptrace""goa.design/clue/log")// ...httpc:=&http.Client{Transport:log.Client(otelhttp.NewTransport(http.DefaultTransport,otelhttp.WithClientTrace(func(ctx context.Context)*httptrace.ClientTrace {returnotelhttptrace.NewClientTrace(ctx)            }),        ),    ),}

Similarly, gRPC clients can be instrumented using the Cluelog and OpenTelemetryotelgrpc packages.

import ("context""go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc""goa.design/clue/log")// ...grpcconn,err:=grpc.DialContext(ctx,"localhost:8080",grpc.WithUnaryInterceptor(log.UnaryClientInterceptor()),grpc.WithStatsHandler(otelgrpc.NewClientHandler())))

Clue relies on the OpenTelemetry API for creating custom instrumentation. Thefollowing snippet illustrates how to create a custom counter and span:

// ... configure OpenTelemetry like in example aboveclue.ConfigureOpenTelemetry(ctx,cfg)// Create a meter and tracermeter:=otel.Meter("mymeter")tracer:=otel.Tracer("mytracer")// Create a countercounter,err:=meter.Int64Counter("my.counter",metric.WithDescription("The number of times the service has been called"),metric.WithUnit("{call}"))iferr!=nil {log.Fatalf("failed to create counter: %s",err)}handler:=http.HandlerFunc(func(w http.ResponseWriter,r*http.Request) {// Create a spanctx,span:=tracer.Start(r.Context(),"myhandler")deferspan.End()// ... do something// Add custom attributes to span and counterattr:=attribute.Int("myattr",42)span.SetAttributes(attr)counter.Add(ctx,1,metric.WithAttributes(attr))// ... do something elseif_,err:=w.Write([]byte("Hello, World!"));err!=nil {log.Errorf(ctx,err,"failed to write response")    }})

Thelog package provides a Goa endpoint middleware that adds the service andmethod names to the logger context. Thedebug package provides a Goa endpointmiddleware that logs the request and response payloads when debug logging isenabled. The example below is a snippet extracted from themain function ofthegenforecasterservice:

svc:=forecaster.New(wc)endpoints:=genforecaster.NewEndpoints(svc)endpoints.Use(debug.LogPayloads())endpoints.Use(log.Endpoint)

Theweather example illustrates how to use Clue to instrumenta system of Goa microservices. The example comes with a set of scripts that canbe used to install all necessary dependencies including theSigNoz instrumentation backend. See theREADME for more information.

The v1.0.0 release ofclue is a major release that introduces breakingchanges. The following sections describe the changes and how to migrate.

Theclue package provides a cohesive API for both metrics and tracing,effectively replacing the previousmetrics andtrace packages. ThetraditionalContext function, utilized in themetrics andtrace packagesfor setting up telemetry, has been deprecated. In its place, theclue packageintroduces theNewConfig function, which generates aConfig object used inconjunction with theConfigureOpenTelemetry function to facilitate telemetrysetup.

v0.x:

ctx:=log.Context(context.Background())traceExporter:=tracestdout.New()metricsExporter:=metricstdout.New()ctx=trace.Context(ctx,"service",traceExporter)ctx=metrics.Context(ctx,"service",metricsExporter)

v1.x:

ctx:=log.Context(context.Background())traceExporter:=tracestdout.New()metricsExporter:=metricstdout.New()cfg:=clue.NewConfig(ctx,"service","1.0.0",metricsExporter,traceExporter)clue.ConfigureOpenTelemetry(ctx,cfg)

Instrumenting a HTTP service is now done using the standardotelhttp package:

v0.x:

handler=trace.HTTP(ctx)(handler)handler=metrics.HTTP(ctx)(handler)http.ListenAndServe(":8080",handler)

v1.x:

http.ListenAndServe(":8080",otelhttp.NewHandler("service",handler))

Similarly, instrumenting a gRPC service is now done using the standardotelgrpc package. v1.x also switches to using a gRPC stats handler instead ofinterceptors:

v0.x:

grpcsvr:=grpc.NewServer(grpcmiddleware.WithUnaryServerChain(trace.UnaryServerInterceptor(ctx),metrics.UnaryServerInterceptor(ctx),        ),grpcmiddleware.WithStreamServerChain(trace.StreamServerInterceptor(ctx),metrics.StreamServerInterceptor(ctx),        ),)

v1.x:

grpcsvr:=grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))

Theotel Goa plugin leverages the OpenTelemetry API to annotate spans andmetrics with HTTP routes.

v0.x:

mux:=goahttp.NewMuxer()ctx=metrics.Context(ctx,genfront.ServiceName,metrics.WithRouteResolver(func(r*http.Request)string {returnmux.ResolvePattern(r)        }),)

v1.x:

package designimport (        ."goa.design/goa/v3/dsl"        _"goa.design/plugins/v3/clue")

SeeContributing