We are currently experiencing degraded performance affecting a subset of users utilizing our REST API for retrieving runs and events data. Responses may be delayed or temporarily return incomplete data, with recent updates not appearing immediately. We have identified the cause of the issue. We're actively working on implementing a fix to resume normal operation of the REST API. Function execution is not affected and is working as expected.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational. Related to the Vercel incident, we updated our dashboard to Node 22.x to solve the issue. We continue to monitor Vercel's incident and react accordingly.

https://www.vercel-status.com/incidents/5r9bp5y8rql2

The incident is now resolved and the system is full operational.

During this incident step.fetch and step.ai.infer were failing due to a bug causing empty request bodies to be returned. The root cause was determined, the system was rolled back and a fix will be rolled out today.

The incident is now resolved and the system is full operational. This was related to an issue caused by the part of the system powering the debounce feature. The internal event backlog is fully caught up and the two mitigations deployed have addressed the issue. The team is preparing a post-mortem to ensure this issue does not reoccur.

After an extended monitoring period, we are resolving this incident. The system is full operational.

The networking fix has been applied and all systems are operational.
We have identified the root cause. Function execution has returned to normal. Any backlogs incurred during the incident will be executed.

The incident is now resolved and the system is full operational after monitoring.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

After an extended monitoring period, function execution has returned to normal rates across all queue shards. During this issue, only a subset of users were affected on part of our infrastructure. Our infrastructure team is in the midst of rolling out additional system capacity going forward.

Runs, traces and events data are all caught up from their temporary backlog. The dashboard metrics are all being processed with no backlog.

System latency for function execution has returned to normal levels for the affected users. The incident has been resolved.

The cause of the incident was due to increased load causing congestion. We applied changes to the system to reduce congestion, resulting in increasing throughput. We also re-distributed some affected users in an effort to mitigate impact.

Our team's planned to roll out new infrastructure in the coming weeks and is accelerating that plan to aim to roll it out later this week to increase overall capacity.

We shipped a fix earlier today at 20:40 UTC (Jan 27) that has resolved the issue after an extended monitoring window to confirm the issue did not return.

The bug itself affecting some user requests was an issue due to a large "baggage" header which was beyond the limit of what the Event API's pubsub event stream can handle. Baggage headers are used for sending extra context within a request like open telemetry or APM tracing data. Some requests contained more than 1024 bytes which caused this issue. The fix applied earlier today gracefully now gracefully handles the situation where there are large baggage headers. This issue will not surface again.