Writes direct to Elasticsearch using the [Elastic Common Schema (ECS)]Elastic Common Schema (ECS)), with semantic logging of structured data from message and scope values. The results can be viewed and queried in the Kibana console.

Add a reference to theElastic.Extensions.Logging package:

<PackageReference Include="Elastic.Extensions.Logging" Version="8.6.0" />

Then, add the provider to the loggingBuilder during host construction, using the provided extension method.

using Elastic.Extensions.Logging;// ...    .ConfigureLogging((hostContext, loggingBuilder) =>    {        loggingBuilder.AddElasticsearch();    })

The default configuration will write to a local Elasticsearch running athttp://localhost:9200/.

Once you have sent some log event, open Kibana (e.g.http://localhost:5601/) and define an index pattern for "dotnet-*" with the time filter "@timestamp".

You can then discover the log events for the index. Some useful columns to add arelog.level,log.logger,event.code,message,tags, andprocess.thread.id.

If you are running multiple applications or on multiple servers, you might want to includeservice.type,service.version, andhost.hostname.

Additional fields are defined below, and all individual message and scope values are logged aslabels.* custom key/value pairs, e.g.labels.CustomerId.

For deployment you will usually want to override the configuration with your actual server location(s). The other useful value to configure is a tag for the environment, e.g. Development/Staging/Production.

{  "Logging": {    "Elasticsearch": {      "NodeUris": [ "https://elastic-staging.example.com:9200" ],      "Tags": [ "Staging" ]    }  }}

NOTE: You don’t need any configuration to just use a local Elasticsearch instance, as it defaults tohttp://localhost:9200/.

The logger provider will be automatically configured with any logging settings under the aliasElasticsearch.

The following default settings are used.

{  "Logging": {    "Elasticsearch": {      "IncludeHost": true,      "IncludeProcess": true,      "IncludeScopes": true,      "IncludeUser": true,      "Index": "dotnet-{0:yyyy.MM.dd}",      "IndexOffset": null,      "IsEnabled": true,      "ListSeparator": ", ",      "MapCorrelationValues": true,      "Tags": [],      "ShipTo": {        "NodePoolType": "SingleNode",        "NodeUris": [ "http://localhost:9200" ]      }    }  }}

The ShipTo settings can have the following properties, depending on the type of connection pool.

If you want to configure from a different section, it can be configured manually:

.ConfigureLogging((hostContext, loggingBuilder) =>{    loggingBuilder.AddElasticsearch(options =>        hostContext.Configuration.Bind("Logging:CustomElasticsearch", options));})

Configuration can, of course, also be done in code, e.g. to add the environment as a tag.

IfCloudId is provided, theConnectionPoolType defaults toCloud:

{  "Logging": {    "Elasticsearch": {      "ShipTo": {        "CloudId": "12345",        "ApiKey": "abcdef"      }    }  }}

Log messages sent to Elasticsearch follow the [Elastic Common Schema (ECS)]Elastic Common Schema (ECS)).

The_source field is the message sent from the LoggerProvider, along with the_index and_id (a GUID).

{  "_index": "dotnet-2020.04.12",  "_type": "_doc",  "_id": "563503a8-9d10-46ff-a09f-c6ccbf124db9",  "_version": 1,  "_score": null,  "_source": {    "MessageTemplate": "Unexpected error processing customer {CustomerId}.",    "Scopes": [      "IP address 2001:db8:85a3::8a2e:370:7334",      "PlainScope"    ],    "agent": {      "version": "1.0.0+bd3ad6",      "type": "Elastic.Extensions.Logging.LoggerProvider"    },    "ecs": {      "version": "1.5.0"    },    "error": {      "message": "Calculation error",      "type": "System.Exception",      "stack_trace": "System.Exception: Calculation error\n ---> System.DivideByZeroException: Attempted to divide by zero.\n   at HelloElasticsearch.Worker.ExecuteAsync(CancellationToken stoppingToken) in /home/sly/Code/essential-logging/examples/HelloElasticsearch/Worker.cs:line 80\n   --- End of inner exception stack trace ---\n   at HelloElasticsearch.Worker.ExecuteAsync(CancellationToken stoppingToken) in /home/sly/Code/essential-logging/examples/HelloElasticsearch/Worker.cs:line 84"    },    "event": {      "code": "5000",      "action": "ErrorProcessingCustomer",      "severity": 3    },    "host": {      "os": {        "platform": "Unix",        "full": "Linux 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020",        "version": "4.15.0.91"      },      "hostname": "VUB1804",      "architecture": "X64"    },    "log": {      "level": "Error",      "logger": "HelloElasticsearch.Worker"    },    "process": {      "thread": {        "id": 10      },      "pid": 25982,      "name": "HelloElasticsearch"    },    "service": {      "type": "HelloElasticsearch",      "version": "1.0.0"    },    "user": {      "id": "sgryphon+es@live.com",      "name": "sly",      "domain": "VUB1804"    },    "@timestamp": "2020-04-13T21:25:22.3352989+10:00",    "tags": [      "Development"    ],    "labels": {      "ip": "2001:db8:85a3::8a2e:370:7334",      "CustomerId": "12345"    },    "message": "Unexpected error processing customer 12345.",    "trace": {      "id": "c20bde1071f7cf4e9a6f368c824e05f7"    },    "transaction": {      "id": "92ba5ee64d963746"    }  },  "fields": {    "@timestamp": [      "2020-04-13T11:25:22.335Z"    ]  },  "sort": [    1586777122335  ]}

Theevent.severity field is numeric and can be used to order events by level, e.g. Kibana queryevent.severity <= 4 will get all messages that have log levelWarning or worse.

If the log message includes an exception, the details are reported in the error fields.

Following the ECS conventions, these use alternative Title casing, to identify them as non-standard fields.

Label values can be accessed by their key, for example if the message, or scope, includes a parameter CustomerId, then the value will be logged aslabels.CustomerId can be searched in Kibana using "labels.CustomerId: 12345".

Example:

The following will generate two labels,labels.EndTime from the message andlabels.CustomerId from the scope:

using (_logger.BeginScope("{CustomerId}", customerId)){  _logger.LogWarning("End of processing reached at {EndTime}.", end);}

Labels are taken from the message (state) and any scope values (may be disabled via the configuration options). In Microsoft.Extensions.Logging an internal FormattedLogValues is used in the ILogger overloads for log levels and scopes; it implements theIEnumerable<KeyValuePair<string,object>> interface that is used to extract the individual parameter values.

Thelabels property in ECS should not contain nested objects, so values are converted to keyword strings. For most objects this is just calling ToString(), with specific formats for some types, e.g. calling string on a list is usually not very useful, so the contents of the list is logged instead.

Labels value formatting

**(1) Seehttps://docs.microsoft.com/en-us/dotnet/standard/datetime/choosing-between-datetime**

These identify the version of the logger provider being used.

This identifies the application/service that is running and generating the logs.

The values are pulled from the entry assemb,Assembly.GetEntryAssembly(), using theName andAssemblyInformationalVersionAttribute values (if informational version is not set it falls back to assemblyVersion).

Note: You should be using a build process that sets the assembly informational version correctly. e.g. If you have a dotnet project using git you can install the local toolGitVersion.Tool, and use it to automatically generate semantic version numbers from the git branch information.

To install the tool:

dotnet new tool-manifestdotnet tool install GitVersion.Tool

Then use the tool to create a semantic version number that can be used in your build process:

dotnet tool restoredotnet gitversion

ASP.NET will automatically pass correlation identifiers between tiers; from 3.0 it also supports the W3C Trace Context standard (https://www.w3.org/TR/trace-context/).

The value ofActivity.Current.RootId is used as the cross-service identifier (in W3C format this is the Trace ID), if in W3C format the Span ID portion ofActivity.Current.Id is used for the transaction, otherwise the full value is used (this is consistent with the way ASP.NET works).

It is recommended to turn on W3C format, for compatibility with other systems:

Activity.DefaultIdFormat = ActivityIdFormat.W3C;