Resolved -As of 11:20am ET, both the Currents and Braze CDI services have been fully restored to normal operational performance. Since that time, all CDI syncs and new Currents events have been processing within standard parameters. The Braze Engineering team has completed the immediate restoration work and is continuing to process the backlog of Currents events produced during the incident's impact window. Depending on the volume of events and the downstream destination for your integration, the backlog processing may take some time and will be different for each integration. As such, with systems operating normally for new incoming events, we are marking this issue as resolved at this time. Should you have questions, please reach out to Braze Support.
Mar10,18:21 EDT
Update -We are continuing to monitor for any further issues.
Mar10,15:01 EDT
Monitoring -Currents is functioning normally, with all new events processing within normal operational bounds. Our Braze Engineering team is actively working to clear the backlog of Currents events.
Mar10,14:48 EDT
Update -Outbound messaging continues to operate within normal bounds.
The Braze CDI continues to be fully operational, and all new events are being processed within normal operational bounds.
The Currents Service has been restored, and new events are being processed within normal operational bounds. Braze Engineers continue to work on processing the backlog of events impacted by the incident.
Mar10,12:35 EDT
Update -Outbound Messaging is continuing to operate within normal bounds.
The Braze CDI is now fully operational, and all new events are being processed within normal operational bounds. If you do not regularly sync the CDI integration, you will need to replay any syncs that failed to fully complete. This step is necessary to prevent old data from overwriting new data that was successfully synced during the incident. You can check the status of your jobs in the dashboard under the Sync log tab within the Cloud Data Ingestion page.
Our Braze Engineers are actively working to restore the Currents service. We will provide further updates as the restoration progresses.
Mar10,11:47 EDT
Update -Outbound Messaging is operating within normal bounds following the changes deployed by The Braze Engineers. At this time Currents and CDI Services latency remains very high and our Engineers are continuing to work to restore the Currents and CDI Services.
Mar10,08:56 EDT
Update -The Braze Engineers have deployed changes to decrease Messaging latency and most messages are sent within normal operational bounds. Our Engineers continue to work on restoring the Currents and CDI Services. At this time Currents and CDI Services latency remains very high.
Mar10,07:15 EDT
Update -Braze Engineers are actively working to resolve the ongoing issue impacting Currents and CDI. We are now observing elevated messaging latency as a result of this issue.
Currently, latency remains highly elevated for Currents, and some customers may also experience latency with CDI integrations.
We appreciate your patience as we work to resolve these issues.
Mar10,04:16 EDT
Update -Engineers continue to work to resolve the issue impacting Currents and CDI.
Latency is currently highly elevated for Currents. Additionally, some customers may experience latency with CDI integrations.
Mar10,03:11 EDT
Update -We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
Mar10,02:17 EDT
Update -We continue to observe elevated latency impacting Currents and CDI. Our engineering teams are actively working to resolve this and restore normal functionality as quickly as possible.
Mar10,01:06 EDT
Identified -We are currently experiencing latency issues with Currents and CDI due to an infrastructure problem. Our engineering team is actively working to resolve this and restore normal functionality as quickly as possible.
Mar10,00:27 EDT