- Notifications
You must be signed in to change notification settings - Fork20
🚨 Design workflows of slog handlers: pipeline, middleware, fanout, routing, failover, load balancing...
License
samber/slog-multi
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Design workflows ofslog handlers:
- Fanout: distribute
log.Record
to multipleslog.Handler
in parallel - Pipe: rewrite
log.Record
on the fly (eg: for privacy reasons) - Router: forward
log.Record
to all matchingslog.Handler
- Failover: forward
log.Record
to the first availableslog.Handler
- Pool: increase log bandwidth by sending
log.Record
to a pool ofslog.Handler
- RecoverHandlerError: catch panics and errors from handlers
Here is a simple workflow with both pipeline and fanout:
Middlewares:
- Inline handler: a shortcut to implement
slog.Handler
- Inline middleware: a shortcut to implement
slogmulti.Middleware
Sponsored by:

See also:
- slog-multi:
slog.Handler
chaining, fanout, routing, failover, load balancing... - slog-formatter:
slog
attribute formatting - slog-sampling:
slog
sampling policy - slog-mock:
slog.Handler
for test purposes
HTTP middlewares:
- slog-gin: Gin middleware for
slog
logger - slog-echo: Echo middleware for
slog
logger - slog-fiber: Fiber middleware for
slog
logger - slog-chi: Chi middleware for
slog
logger - slog-http:
net/http
middleware forslog
logger
Loggers:
- slog-zap: A
slog
handler forZap
- slog-zerolog: A
slog
handler forZerolog
- slog-logrus: A
slog
handler forLogrus
Log sinks:
- slog-datadog: A
slog
handler forDatadog
- slog-betterstack: A
slog
handler forBetterstack
- slog-rollbar: A
slog
handler forRollbar
- slog-loki: A
slog
handler forLoki
- slog-sentry: A
slog
handler forSentry
- slog-syslog: A
slog
handler forSyslog
- slog-logstash: A
slog
handler forLogstash
- slog-fluentd: A
slog
handler forFluentd
- slog-graylog: A
slog
handler forGraylog
- slog-quickwit: A
slog
handler forQuickwit
- slog-slack: A
slog
handler forSlack
- slog-telegram: A
slog
handler forTelegram
- slog-mattermost: A
slog
handler forMattermost
- slog-microsoft-teams: A
slog
handler forMicrosoft Teams
- slog-webhook: A
slog
handler forWebhook
- slog-kafka: A
slog
handler forKafka
- slog-nats: A
slog
handler forNATS
- slog-parquet: A
slog
handler forParquet
+Object Storage
- slog-channel: A
slog
handler for Go channels
go get github.com/samber/slog-multi
Compatibility: go >= 1.21
No breaking changes will be made to exported APIs before v2.0.0.
Warning
Use this library carefully, log processing can be very costly (!)
GoDoc:https://pkg.go.dev/github.com/samber/slog-multi
Distribute logs to multipleslog.Handler
in parallel.
import ( slogmulti"github.com/samber/slog-multi""log/slog")funcmain() {logstash,_:=net.Dial("tcp","logstash.acme:4242")// use github.com/netbrain/goautosocket for auto-reconnectstderr:=os.Stderrlogger:=slog.New(slogmulti.Fanout(slog.NewJSONHandler(logstash,&slog.HandlerOptions{}),// pass to first handler: logstash over tcpslog.NewTextHandler(stderr,&slog.HandlerOptions{}),// then to second handler: stderr// ... ), )logger.With(slog.Group("user",slog.String("id","user-123"),slog.Time("created_at",time.Now()), ), ).With("environment","dev").With("error",fmt.Errorf("an error")).Error("A message")}
Stderr output:
time=2023-04-10T14:00:0.000000+00:00 level=ERROR msg="A message" user.id=user-123 user.created_at=2023-04-10T14:00:0.000000+00:00 environment=dev error="an error"
Netcat output:
{"time":"2023-04-10T14:00:0.000000+00:00","level":"ERROR","msg":"A message","user":{"id":"user-123","created_at":"2023-04-10T14:00:0.000000+00:00"},"environment":"dev","error":"an error"}
Distribute logs to all matchingslog.Handler
in parallel.
import ( slogmulti"github.com/samber/slog-multi" slogslack"github.com/samber/slog-slack""log/slog")funcmain() {slackChannelUS:= slogslack.Option{Level:slog.LevelError,WebhookURL:"xxx",Channel:"supervision-us"}.NewSlackHandler()slackChannelEU:= slogslack.Option{Level:slog.LevelError,WebhookURL:"xxx",Channel:"supervision-eu"}.NewSlackHandler()slackChannelAPAC:= slogslack.Option{Level:slog.LevelError,WebhookURL:"xxx",Channel:"supervision-apac"}.NewSlackHandler()logger:=slog.New(slogmulti.Router().Add(slackChannelUS,recordMatchRegion("us")).Add(slackChannelEU,recordMatchRegion("eu")).Add(slackChannelAPAC,recordMatchRegion("apac")).Handler(), )logger.With("region","us").With("pool","us-east-1").Error("Server desynchronized")}funcrecordMatchRegion(regionstring)func(ctx context.Context,r slog.Record)bool {returnfunc(ctx context.Context,r slog.Record)bool {ok:=falser.Attrs(func(attr slog.Attr)bool {ifattr.Key=="region"&&attr.Value.Kind()==slog.KindString&&attr.Value.String()==region {ok=truereturnfalse }returntrue })returnok }}
List multiple targets for aslog.Record
instead of retrying on the same unavailable log management system.
import ("net" slogmulti"github.com/samber/slog-multi""log/slog")funcmain() {// ncat -l 1000 -k// ncat -l 1001 -k// ncat -l 1002 -k// list AZs// use github.com/netbrain/goautosocket for auto-reconnectlogstash1,_:=net.Dial("tcp","logstash.eu-west-3a.internal:1000")logstash2,_:=net.Dial("tcp","logstash.eu-west-3b.internal:1000")logstash3,_:=net.Dial("tcp","logstash.eu-west-3c.internal:1000")logger:=slog.New(slogmulti.Failover()( slog.HandlerOptions{}.NewJSONHandler(logstash1,nil),// send to this instance first slog.HandlerOptions{}.NewJSONHandler(logstash2,nil),// then this instance in case of failure slog.HandlerOptions{}.NewJSONHandler(logstash3,nil),// and finally this instance in case of double failure ), )logger.With(slog.Group("user",slog.String("id","user-123"),slog.Time("created_at",time.Now()), ), ).With("environment","dev").With("error",fmt.Errorf("an error")).Error("A message")}
Increase log bandwidth by sendinglog.Record
to a pool ofslog.Handler
.
import ("net" slogmulti"github.com/samber/slog-multi""log/slog")funcmain() {// ncat -l 1000 -k// ncat -l 1001 -k// ncat -l 1002 -k// list AZs// use github.com/netbrain/goautosocket for auto-reconnectlogstash1,_:=net.Dial("tcp","logstash.eu-west-3a.internal:1000")logstash2,_:=net.Dial("tcp","logstash.eu-west-3b.internal:1000")logstash3,_:=net.Dial("tcp","logstash.eu-west-3c.internal:1000")logger:=slog.New(slogmulti.Pool()(// a random handler will be picked slog.HandlerOptions{}.NewJSONHandler(logstash1,nil), slog.HandlerOptions{}.NewJSONHandler(logstash2,nil), slog.HandlerOptions{}.NewJSONHandler(logstash3,nil), ), )logger.With(slog.Group("user",slog.String("id","user-123"),slog.Time("created_at",time.Now()), ), ).With("environment","dev").With("error",fmt.Errorf("an error")).Error("A message")}
Returns aslog.Handler
that recovers from panics or error of the chain of handlers.
import (slogformatter"github.com/samber/slog-formatter"slogmulti"github.com/samber/slog-multi""log/slog")recovery:=slogmulti.RecoverHandlerError(func(ctx context.Context,record slog.Record,errerror) {// will be called only if subsequent handlers fail or return an errorlog.Println(err.Error()) },)sink:=NewSinkHandler(...)logger:=slog.New(slogmulti.Pipe(recovery).Handler(sink),)err:=fmt.Errorf("an error")logger.Error("a message",slog.Any("very_private_data","abcd"),slog.Any("user",user),slog.Any("err",err))// outputs:// time=2023-04-10T14:00:0.000000+00:00 level=ERROR msg="a message" error.message="an error" error.type="*errors.errorString" user="John doe" very_private_data="********"
Rewritelog.Record
on the fly (eg: for privacy reason).
funcmain() {// first middleware: format go `error` type into an object {error: "*myCustomErrorType", message: "could not reach https://a.b/c"}errorFormattingMiddleware:=slogmulti.NewHandleInlineMiddleware(errorFormattingMiddleware)// second middleware: remove PIIgdprMiddleware:=NewGDPRMiddleware()// final handlersink:=slog.NewJSONHandler(os.Stderr,&slog.HandlerOptions{})logger:=slog.New(slogmulti.Pipe(errorFormattingMiddleware).Pipe(gdprMiddleware).// ...Handler(sink), )logger.With(slog.Group("user",slog.String("id","user-123"),slog.String("email","user-123"),slog.Time("created_at",time.Now()), ), ).With("environment","dev").Error("A message",slog.String("foo","bar"),slog.Any("error",fmt.Errorf("an error")), )}
Stderr output:
{"time":"2023-04-10T14:00:0.000000+00:00","level":"ERROR","msg":"A message","user":{"id":"*******","email":"*******","created_at":"*******" },"environment":"dev","foo":"bar","error":{"type":"*myCustomErrorType","message":"an error" }}
Middleware must match the following prototype:
typeMiddlewarefunc(slog.Handler) slog.Handler
The example above uses:
Note:WithAttrs
andWithGroup
methods of custom middleware must return a new instance, instead ofthis
.
An "inline handler" (aka. lambda), is a shortcut to implementslog.Handler
, that hooks a single method and proxies others.
mdw:=slogmulti.NewHandleInlineHandler(// simulate "Handle()"func(ctx context.Context,groups []string,attrs []slog.Attr,record slog.Record)error {// [...]returnnil },)
mdw:=slogmulti.NewInlineHandler(// simulate "Enabled()"func(ctx context.Context,groups []string,attrs []slog.Attr,level slog.Level)bool {// [...]returntrue },// simulate "Handle()"func(ctx context.Context,groups []string,attrs []slog.Attr,record slog.Record)error {// [...]returnnil },)
An "inline middleware" (aka. lambda), is a shortcut to implement middleware, that hooks a single method and proxies others.
// hook `logger.Enabled` methodmdw:=slogmulti.NewEnabledInlineMiddleware(func(ctx context.Context,level slog.Level,nextfunc(context.Context, slog.Level)bool)bool{// [...]returnnext(ctx,level)})
// hook `logger.Handle` methodmdw:=slogmulti.NewHandleInlineMiddleware(func(ctx context.Context,record slog.Record,nextfunc(context.Context, slog.Record)error)error {// [...]returnnext(ctx,record)})
// hook `logger.WithAttrs` methodmdw:=slogmulti.NewWithAttrsInlineMiddleware(func(attrs []slog.Attr,nextfunc([]slog.Attr) slog.Handler) slog.Handler{// [...]returnnext(attrs)})
// hook `logger.WithGroup` methodmdw:=slogmulti.NewWithGroupInlineMiddleware(func(namestring,nextfunc(string) slog.Handler) slog.Handler{// [...]returnnext(name)})
A super inline middleware that hooks all methods.
Warning: you would rather implement your own middleware.
mdw:=slogmulti.NewInlineMiddleware(func(ctx context.Context,level slog.Level,nextfunc(context.Context, slog.Level)bool)bool{// [...]returnnext(ctx,level) },func(ctx context.Context,record slog.Record,nextfunc(context.Context, slog.Record)error)error{// [...]returnnext(ctx,record) },func(attrs []slog.Attr,nextfunc([]slog.Attr) slog.Handler) slog.Handler{// [...]returnnext(attrs) },func(namestring,nextfunc(string) slog.Handler) slog.Handler{// [...]returnnext(name) },)
- Ping me on twitter@samuelberthe (DMs, mentions, whatever :))
- Fork theproject
- Fixopen issues or request new features
Don't hesitate ;)
# Install some dev dependenciesmake tools# Run testsmaketest# ormake watch-test
Give a ⭐️ if this project helped you!
Copyright © 2023Samuel Berthe.
This project isMIT licensed.
About
🚨 Design workflows of slog handlers: pipeline, middleware, fanout, routing, failover, load balancing...