Logging on Toolforge, both for user workloads and for the infrastructure, is being moved to a setup based onGrafana Loki.

Log storage is handled byGrafana Loki, with persistant storage in the Ceph cluster via theS3 interface. The s3 buckets exist in separate projects,tools-logging andtoolsbeta-logging, as our RadosGW implementation does not allow for more specific than per-project access control restrictions. The buckets are created viatofu-provisioning system.

The different Loki deployments in each project (tools and toolsbeta) are:

tools

Log storage fortool workloads. (So everything in an individualtool- namespace.)

EachKubernetes worker node runs aGrafana Alloy pod that forwards logs from pods running on that node to the appropriate Loki instance.

The entire logging stack is deployed via thelogging component oftoolforge-deploy.

AGrafana dashboard is available.

• Help:Toolforge/Logs

• Toolforge admin