The issue affecting wait for event steps not resolving has been resolved. The investigation indicated that the errors started on October 9th around 17:43 UTC and was resolved at October 10th at 16:11 UTC.
The issue only affected customers that were recently migrated to new queueing infrastructure on October 9th. These customers include a subset of our free and paid users, but Enterprise users were unaffected. We apologize for the issue and a post mortem will be completed as soon as possible.
Any function runs affected by this issue may appear as stuck in the "Running" status. These runs can be cancelled or re-run.
Resolved
The issue affecting wait for event steps not resolving has been resolved. The investigation indicated that the errors started on October 9th around 17:43 UTC and was resolved at October 10th at 16:11 UTC.
The issue only affected customers that were recently migrated to new queueing infrastructure on October 9th. These customers include a subset of our free and paid users, but Enterprise users were unaffected. We apologize for the issue and a post mortem will be completed as soon as possible.
Any function runs affected by this issue may appear as stuck in the "Running" status. These runs can be cancelled or re-run.
Monitoring
The team has deployed a fix for this issue and wait for event should be running as of normal after 16:11 UTC. Newly sent events should resolve wait for event steps after that time. Previous steps may be stuck and be required to be cancelled and re-ran.
Identified
We have identified the cause of the issue related to new infrastructure recently rolled out. We're actively working on implementing a fix to resume normal operation of the system.
Investigating
We are actively investigating an issue with wait for event steps that is causing delays affecting several customers. We will provider further updates as we identify the cause and resolve the issue.