The incident is now resolved and the system is fully operational.

Performance on all shards is back to normal levels. The degradation was caused by a change that aimed to improve concurrency metrics for Inngest accounts. The change, while out for a full 24 hours and fairly benign, began to compound and produced slowness for some queue shards within our system earlier today. We have reverted that change and are working to understand the performance impact of this change.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational. Related to the Vercel incident, we updated our dashboard to Node 22.x to solve the issue. We continue to monitor Vercel's incident and react accordingly.

https://www.vercel-status.com/incidents/5r9bp5y8rql2

The incident is now resolved and the system is full operational.

During this incident step.fetch and step.ai.infer were failing due to a bug causing empty request bodies to be returned. The root cause was determined, the system was rolled back and a fix will be rolled out today.

The incident is now resolved and the system is full operational. This was related to an issue caused by the part of the system powering the debounce feature. The internal event backlog is fully caught up and the two mitigations deployed have addressed the issue. The team is preparing a post-mortem to ensure this issue does not reoccur.

After an extended monitoring period, we are resolving this incident. The system is full operational.

The networking fix has been applied and all systems are operational.
We have identified the root cause. Function execution has returned to normal. Any backlogs incurred during the incident will be executed.

The incident is now resolved and the system is full operational after monitoring.

The incident is now resolved and the system is full operational.