The incident is now resolved and the system is full operational.

Delays began around 14:03 UTC caused by an issue publishing to a new Kafka topic. Recovery began around 14:40 UTC as the team was able to isolate this topic. The consuming services which schedules new function runs was scaled out and began to consume the backlog. No events were dropped during this incident, all functions were scheduled, but with delays during this time window. No action is needed for manual recovery.

Metrics in the UI are now restored. Users may notice metrics data continue to be unavailable during the time of the incident, from the timeframe 7/16/26 15:45 UTC until approximately 7/16/26 21:07 UTC

Throughput and latency have recovered and the system should be fully operational again.

The core issue is related to the part of the system that persists run state. Some operations experienced and increase in errors and retries, causing a backlog in some parts of the system and some failed operations like checkpoints or signals. We are continuing a thorough post-mortem investigation to prevent reoccurrance.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

The Support Portal is now available.

The incident is now resolved.

The previous incident with the observability database has beens resolved, but there is a separate incident with a downstream provider.

The incident is now resolved and the system is fully operational.

Performance on all shards is back to normal levels. The degradation was caused by a change that aimed to improve concurrency metrics for Inngest accounts. The change, while out for a full 24 hours and fairly benign, began to compound and produced slowness for some queue shards within our system earlier today. We have reverted that change and are working to understand the performance impact of this change.

The incident is now resolved and the system is full operational.