Repository files navigation

SerilogTracing is a minimal tracing system that integrates Serilog with .NET'sSystem.Diagnostics.Activity. You can use it to add distributed, hierarchical tracing to applications that use Serilog, and to consume traces generated by .NET components includingHttpClient and ASP.NET Core.

Traces are written to standard Serilog sinks.Sinks with capable back-ends support full hierarchical tracing, and others will flatten traces into individual spans with timing information.

Here's the output of the includedexample application in the standardSystem.Console sink:

The same trace displayed in Seq:

And in Zipkin:

This section walks through a very simple SerilogTracing example. To get started we'll create a simple .NET 8 console application and install some SerilogTracing packages.

mkdir examplecd exampledotnet new consoledotnet add package SerilogTracingdotnet add package SerilogTracing.Expressionsdotnet add package Serilog.Sinks.Console

Replace the contents of the generatedProgram.cs with:

usingSerilog;usingSerilog.Templates.Themes;usingSerilogTracing;usingSerilogTracing.Expressions;Log.Logger=newLoggerConfiguration().WriteTo.Console(Formatters.CreateConsoleTextFormatter(TemplateTheme.Code)).CreateLogger();usingvarlistener=newActivityListenerConfiguration().TraceToSharedLogger();usingvaractivity=Log.Logger.StartActivity("Check {Host}","example.com");try{varclient=newHttpClient();varcontent=awaitclient.GetStringAsync("https://example.com");Log.Information("Content length is {ContentLength}",content.Length);activity.Complete();}catch(Exceptionex){activity.Complete(LogEventLevel.Fatal,ex);}finally{awaitLog.CloseAndFlushAsync();}

Running it will print some log events and spans to the console:

dotnet run

Let's break the example down a bit.

The Serilog pipeline is set up normally:

usingSerilog;usingSerilog.Templates.Themes;usingSerilogTracing;usingSerilogTracing.Expressions;Log.Logger=newLoggerConfiguration().WriteTo.Console(Formatters.CreateConsoleTextFormatter(TemplateTheme.Code)).CreateLogger();

TheFormatters.CreateConsoleTextFormatter() function comes fromSerilogTracing.Expressions; you can ignore this and use a regular console output template, but the one we're using here produces nice output for spans that includes timing information. Dig into the implementation of theCreateConsoleTextFormatter() function if you'd like to see how to set up your own trace-specific formatting, it's pretty straightforward.

This line sets up SerilogTracing's integration with .NET's diagnostic sources, and starts an activity listener in the background that will write spans from the framework and third-party libraries through your Serilog pipeline:

usingvarlistener=newActivityListenerConfiguration().TraceToSharedLogger();

This step is optional, but you'll need this if you want to view your SerilogTracing output as hierarchical, distributed traces: without it,HttpClient won't generate spans, and won't propagate trace ids along with outbound HTTP requests.

You can also configure SerilogTracing to send spans through a specificILogger:

usingSerilog;usingSerilogTracing;usingSerilogTracing.Expressions;awaitusingvarlogger=newLoggerConfiguration().WriteTo.Console(Formatters.CreateConsoleTextFormatter()).CreateLogger();usingvarlistener=newActivityListenerConfiguration().TraceTo(logger);

ILogger.StartActivity() is the main SerilogTracing API for starting activities. It works on anyILogger, and the span generated by the activity will be written through that logger, receiving the same enrichment and filtering as any other log event.

usingvaractivity=Log.Logger.StartActivity("Check {Host}","example.com");

StartActivity accepts amessage template, just like Serilog, and you can capture structured properties by including them in the template.

The object returned fromStartActivity() is aLoggerActivity, to which you can add additional structured data usingAddProperty().

TheLoggerActivity implementsIDisposable, and if you let the activity be disposed normally, it will record the activity as complete, and write a span through the underlyingILogger.

In the example, because the activity needs to be completed before theLog.CloseAndFlushAsync() call at the end, we callComplete() explicitly on the success path:

try{// ...activity.Complete();}catch(Exceptionex){activity.Complete(LogEventLevel.Fatal,ex);}

On the failure path, we call the overload ofComplete() that accepts a level and exception, to mark the activity as failed and use the specified level for the generated log event.

These sinks have been built or modified to work well with tracing back-ends:

• Serilog.Sinks.Seq - callWriteTo.Seq() to send logs and traces to Seq; useEnrich.WithProperty("Application", "your app") to show service names in traces.
• Serilog.Sinks.OpenTelemetry — callWriteTo.OpenTelemetry() to send traces and logs using OTLP.
• SerilogTracing.Sinks.Zipkin - callWriteTo.Zipkin() to send traces to Zipkin; logs are ignored by this sink.

To add tracing support to an existing sink, seehow activities are mapped ontoLogEvents.

If you're writing an ASP.NET Core application, you'll notice that the spans generated in response to web requests have very generic names, likeHttpRequestIn. To fix that, first addSerilogTracing.Instrumentation.AspNetCore:

dotnet add package SerilogTracing.Instrumentation.AspNetCore --prerelease

Then addInstrument.AspNetCoreRequests() to yourActivityListenerConfiguration:

usingvarlistener=newActivityListenerConfiguration().Instrument.AspNetCoreRequests().TraceToSharedLogger();

HTTP requests received by ASP.NET Core may contain a header with the trace id, span id, and sampling decision made for the active span in the calling application. How this header is used can be configured withHttpRequestInActivityInstrumentationOptions.IncomingTraceParent:

usingvarlistener=newActivityListenerConfiguration().Instrument.AspNetCoreRequests(opts=>{opts.IncomingTraceParent=IncomingTraceParent.Trust;}).TraceToSharedLogger();

The supported options are:

• IncomingTraceParent.Accept (default) — the parent's trace and span ids will be used, but the sampling decision will be ignored; this reveals the presence of incoming tracing information while preventing callers from controlling whether data is recorded
• IncomingTraceParent.Ignore — no information about the parent span will be preserved; this is the appropriate option for most public or Internet-facing sites and services
• IncomingTraceParent.Trust — use the parent's trace and span ids, and respect the parent's sampling decision; this is the appropriate option for many internal services, since it allows system-wide sampling and consistent, detailed traces

See the sectionSampling below for more information on how sampling works in SerilogTracing.

HttpClient requests are instrumented by default. To configure the wayHttpClient requests are recorded as spans, remove the default instrumentation and addHttpClient instrumentation explicitly:

usingvarlistener=newActivityListenerConfiguration().Instrument.WithDefaultInstrumentation(false).Instrument.HttpClientRequests(opts=>opts.MessageTemplate="Hello, world!").TraceToSharedLogger();

The message template for spans, and mappings fromHttpRequestMessage andHttpResponseMessage into log event properties and the completion level can be configured.

Microsoft's client library for SQL Server doesn't generate spans by default. To turn on tracing of database commands, installSerilogTracing.Instrumentation.SqlClient:

dotnet add package SerilogTracing.Instrumentation.SqlClient --prerelease

Then addInstrument.SqlClientCommands() to yourActivityListenerConfiguration:

usingvarlistener=newActivityListenerConfiguration().Instrument.SqlClientCommands().TraceToSharedLogger();

Npgsql is internally instrumented usingSystem.Diagnostics.Activity, so no additional packages or steps are required to enable instrumentation of Npgsql commands. If you're missing spans from Npgsql, check that the"Npgsql" namespace isn't suppressed by yourMinimumLevel.Override() configuration.

SerilogTracing includes extensions toSerilog.Expressions aimed at producing useful text and JSON output fromspans:

dotnet add package SerilogTracing.Expressions --prerelease

For console output,Formatters.CreateConsoleTextFormatter() provides span timings in a pleasant ANSI-colored format:

Log.Logger=newLoggerConfiguration()// The `Formatters` class is from `SerilogTracing.Expressions`.WriteTo.Console(Formatters.CreateConsoleTextFormatter(TemplateTheme.Code)).CreateLogger();

Alternatively,TracingNameResolver can be used withExpressionTemplate to create text or JSON output. Theexample above expands into the (admittedly quite dense) template below:

varformatter=newExpressionTemplate("[{@t:HH:mm:ss} {@l:u3}] "+"{#if IsRootSpan()}\u2514\u2500 {#else if IsSpan()}\u251c {#else if @sp is not null}\u2502 {#else}\u250A {#end}"+"{@m}"+"{#if IsSpan()} ({Milliseconds(Elapsed()):0.###} ms){#end}"+"\n"+"{@x}",theme:TemplateTheme.Code,nameResolver:newTracingNameResolver());Log.Logger=newLoggerConfiguration().WriteTo.Console(formatter).CreateLogger();

For an example showing how to produce JSON withExpressionTemplate, see the implementation ofZipkinSink in this repository,andthis article introducingSerilog.Expressions JSON support.

Sampling is a method of reducing stored data volumes by selectively recording traces. This is similar to leveling, but instead of turning individual span types on and off, sampling causes eitherall of the spans in a trace to be recorded, ornone of them.

SerilogTracing implements two simple strategies viaActivityListenerConfiguration:Sample.AllTraces(), which records all traces (the default), andSample.OneTraceIn(), which records a fixed proportion of possible traces:

// Record only every 1000th traceusingvarlistener=newActivityListenerConfiguration().Sample.OneTraceIn(1000).TraceToSharedLogger();

More sophisticated sampling strategies can be plugged in throughSample.Using(). These behave like the rawSystem.Diagnostics.ActivityListener API, but only apply to root spans. Setting theignoreParent method parameter totrue can be used to exactly mimic theSystem.Diagnostics.ActivityListener behavior.

Sampling does not affect the recording of log events: log events written during an unsampled trace will still be recorded, and will carry trace and span ids even though the corresponding spans will be missing.

Traces sometimes need to cross process or temporal boundaries. For example, the parent of a server activity for a remote procedure call might be a client activity on a remote machine.

If you're usingHttpClient and ASP.NET Core, .NET will do this automatically using thetraceparent HTTP header. If you're in a situation without built-in trace propagation, you'll need to do this yourself, using theStartActivity() overload that accepts anActivityContext:

// Parse a W3C `traceparent` header; alternatively, call the `ActivityContext` constructor// that accepts a trace and span id directly.varcontext=ActivityContext.Parse("00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01",null);usingvaractivity=Log.Logger.StartActivity(ActivityKind.Server,context,LogEventLevel.Information,"Handle request");

Applications using SerilogTracing add tracing usingILogger.StartActivity(). These activities are always converted intoLogEvents and emitted through the originalILogger that created them..NET libraries and frameworks add tracing usingSystem.Diagnostics.ActivitySources. These activities are also emitted asLogEvents when usingSerilogTracing.ActivityListenerConfiguration.

Traces are collections of spans, connected by a common trace id. SerilogTracing maps the typical properties associated with a span onto SerilogLogEvent instances:

SerilogTracing can consume activities from .NET itself, and libraries that don't themselves use SerilogTracing. By default, you'll see spans for all activities, from all sources, in your Serilog output.

To "turn down" the level of tracing performed by an external activity source, use SerilogTracing'sInitialLevel configuration to set a level for spans from that source:

.InitialLevel.Override("Npgsql",LogEventLevel.Debug)

In this example, when activities from theNpgsql activity source are assigned an initial level ofDebug, they'll be suppressed unless your Serilog logger has debug logging enabled.

The initial level assigned to a source determines whether activities are created by the source. When the activity is completed, it may be recorded at a higher level; for example, a span created at an initialInformation level may complete as anError (but not at a lower level such asDebug, because doing so may suppress the span cause the trace hierarchy to become incoherent).

Activities produced by external .NET libraries may include one or more embeddedActivityEvents. By default, SerilogTracingignores these, except in the case ofexception events, which map to theLogEvent.Exception property.

To emit additionalLogEvents for each embeddedActivityEvent, callActivityEvents.AsLogEvents() onActivityListenerConfiguration.

OpenTelemetry is a project that combines a variety of telemetry data models, schemas, APIs, and SDKs. SerilogTracing, like Serilog itself, has no dependency on the OpenTelemetry SDK, but can output traces using the OpenTelemetry Protocol (OTLP). From the point of view of SerilogTracing, this is considered to be just one of many protocols and systems that exist in the wider Serilog ecosystem.

If you're working in an environment with deep investment in OpenTelemetry, you might consider using theOpenTelemetry .NET SDK instead of SerilogTracing. If you're seeking lightweight, deliberate instrumentation that has the same crafted feel and tight control offered by Serilog, you're in the right place.

SerilogTracing is an open source (Apache 2.0) project that welcomes your ideas and contributions. It's built by @nblumhardt (also a Serilog maintainer), @liammclennan and @kodraus from Datalust, the company behind Seq.

SerilogTracing is not an official Serilog or Datalust project, but our hope for it is that it can serve as a validation and a basis for deeper tracing support in Serilog in the future.