Motivation / Background

Ref:#50452

This PR adds structured event reporting to Rails. The event reporter, accessible viaRails.event, allows you to report structured events to subscriber(s), and provides mechanisms for adding tags and context to events. While the existingRails.logger is great for producing human-readable logs, the unstructured log lines it generates are difficult for modern data analytics / observability platforms to parse. We'd like to add first-class support for structured events, complementing the existing Rails logger.

Note that events encompass "structured logs", but also "business events", as well as telemetry events such as metrics and logs. The Event Reporter is designed to be a single interface for producing any kind of event in a Rails application.

This is the API we developed for structured logging at Shopify and are using in our monolith.

Detail

Basic Event Reporting

# Simple event with payloadRails.event.notify("user_created",{id:123,name:"John Doe"})# Event object (for schematized events)Rails.event.notify(UserCreatedEvent.new(id:123,name:"John Doe"))

Event Structure

# All events include standardized metadata:{name:"user_created",payload:{id:123,name:"John Doe"},tags:{section:"admin"},context:{request_id:"abc123",user_agent:"Mozilla..."},timestamp:1738964843208679035,# nanosecond precisionsource_location:{filepath:"app/services/user_service.rb",lineno:45,label:"UserService#create"}}

Adding Tags / Context

# Tags - Domain-specific context that nests:Rails.event.tagged("graphql")doRails.event.tagged(section:"admin")doRails.event.notify("user_created",{id:123})# Event includes tags: { graphql: true, section: "admin" }endend# Context - Request/job-scoped metadata:Rails.event.set_context(request_id:"abc123",shop_id:456)Rails.event.notify("user_created",{id:123})# All subsequent events in this request include the context

Debug Mode - Conditional event reporting:

Rails.event.with_debugdoRails.event.debug("sql_query",{sql:"SELECT * FROM users"})# Only reported when debug mode is activeend

Subscriber Interface

EDIT: I've amended this PR to provide some default encoders out of the box, and we may additionally want to provide a default subscriber that converts structured events into unstructured log lines.

We separate the emission of events from how these events reach end consumers; applications are expected to define their own subscribers, and the Event Reporter is responsible for emitting events to these subscribers.

classMySubscriberdefemit(event)# Process the structured event hash# Serialize and export to logging pipelineendendRails.event.subscribe(MySubscriber.new)

Testing Support

# Assert specific events are reportedassert_event_reported("user.created",payload:{id:123})doUserService.create_user(name:"John")end# Assert no events are reportedassert_no_event_reported("user.deleted")doUserService.update_user(id:123,name:"Jane")end

Key Design Decisions

1. Fiber-based Isolation: Tags and context are scoped per fiber. Child fibers / threads inherit context from the parent, but the context is copied on write so that changes do not propagate to the parent.
2. Structured Metadata: Every event includes consistent name, timestamp, source location, payload, tags, and context.
3. Flexible Payloads: Supports both hash payloads and custom event objects for implicitly structured events and explicitly schematized events.
4. Publisher-Subscriber: Decouples event emission from processing, allowing multiple subscribers and providing flexibility for applications to define how events reach consumers.

Additional information

What is an event?

"Business Events", Traces, Spans, Metrics and Logs are all types of events. At Shopify, we built the Structured Event Reporter to be a unified solution for any and all of these use cases.

Why support arbitrarily structured payloads (hashes) and "event objects"?

It is important for the API to support both implicitly structured events (hashes) and explicitly typed events that comply with a schema. At Shopify, we're moving towards a future where all our events are schematized. The Event Reporter allows "event objects" to be passed to the Event Reporter, and these can have defined schemas. These objects are passed directly to subscribers in thepayload field; we don't make any assumptions about how these should be serialized. The subscriber(s) need to handle encoding these appropriately.

Why no log levels?

Log levels are rather arbitrary. Different logging systems use different levels, and these are interpreted differently depending on the user / context and are generally used inconsistently. The primary use case for log levels is to distinguish:

Things that developers care about when they are developing or debugging software.
Things that users care about when using your software.

Obviously these are debug and info levels, respectively.

(Source:https://dave.cheney.net/2015/11/05/lets-talk-about-logging)

As such, when designing this API, we opted to implicitly support "log levels" in the API viaRails.event.notify andRails.event.debug. We don't require additional level parameters, nor do we expose different levels in the API.

How do the payload, tags, and context differ?

Thepayload, or event object, contains information about an event that occurred, typically specific to a given domain.Tags can be used to enrich an event with further context, and form a "stack"; all events within the block will include tags currently pushed onto the stack. Note that tags and the event payload may occupy logically different domains. Thecontext is designed for request/job-level metadata that spans the entire execution context. Context should expand over the course of the unit of work, and will be attached to every event emitted by the event reporter.

Why are we not usingActiveSupport::ExecutionContext for the event context?

At Shopify, we specifically want to use Fiber storage as the underlying storage mechanism for storing event reporter context. Weexplored making changes to the existing execution context / isolated execution state singletons to allow this, but we decided it made more sense to handroll what we needed specifically for the event reporter, and then revisit folding this back into AS::EC / AS::ISE later.

Checklist

Before submitting the PR make sure the following are checked:

• This Pull Request is related to one change. Unrelated changes should be opened in separate PRs.
• Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex:[Fix #issue-number]
• Tests are added or updated if you fix a bug or add a feature.
• CHANGELOG files are updated for the changed libraries if there is a behavior change or additional feature. Minor bug fixes and documentation changes should not be included.