The incident is now resolved and the system is full operational.

After an extended monitoring period, function execution has returned to normal rates across all queue shards. During this issue, only a subset of users were affected on part of our infrastructure. Our infrastructure team is in the midst of rolling out additional system capacity going forward.

Runs, traces and events data are all caught up from their temporary backlog. The dashboard metrics are all being processed with no backlog.

System latency for function execution has returned to normal levels for the affected users. The incident has been resolved.

The cause of the incident was due to increased load causing congestion. We applied changes to the system to reduce congestion, resulting in increasing throughput. We also re-distributed some affected users in an effort to mitigate impact.

Our team's planned to roll out new infrastructure in the coming weeks and is accelerating that plan to aim to roll it out later this week to increase overall capacity.