The incident is now resolved and the system is full operational. This was related to an issue caused by the part of the system powering the debounce feature. The internal event backlog is fully caught up and the two mitigations deployed have addressed the issue. The team is preparing a post-mortem to ensure this issue does not reoccur.
Resolved
The incident is now resolved and the system is full operational. This was related to an issue caused by the part of the system powering the debounce feature. The internal event backlog is fully caught up and the two mitigations deployed have addressed the issue. The team is preparing a post-mortem to ensure this issue does not reoccur.
Monitoring
The mitigations have fixed the issue and the event backlog is now caught up. We continue to monitor and evaluate other short and long term mitigations to add.
Identified
We have deployed an additional hot fix. The earlier change rolled out have addressed the core issue and the system is now processing the backlog. The backlog is decreasing. We will provide another update as we have an estimate on time to recovery.
Identified
We have identified the cause of the issue affecting a core system queue. We are rolling our a mitigation now and preparing follow up changes.
Investigating
We are actively investigating delays with function run scheduling. We will provider further updates as we identify the cause and resolve the issue.