Performance on all shards is back to normal levels. The degradation was caused by a change that aimed to improve concurrency metrics for Inngest accounts. The change, while out for a full 24 hours and fairly benign, began to compound and produced slowness for some queue shards within our system earlier today. We have reverted that change and are working to understand the performance impact of this change.
Resolved
Performance on all shards is back to normal levels. The degradation was caused by a change that aimed to improve concurrency metrics for Inngest accounts. The change, while out for a full 24 hours and fairly benign, began to compound and produced slowness for some queue shards within our system earlier today. We have reverted that change and are working to understand the performance impact of this change.
Monitoring
Function execution latency has returned to normal for affected customers. Some customer shards were affected and we've pinpointed the cause of a slow degradation that compounded over time. We are working on adding new monitoring to catch performance regressions in this part of the system more quickly.
Investigating
We are actively investigating increased function execution latency on a subset of customer shards starting around 5pm UTC. We will provider further updates as we identify the cause and resolve the issue.
