Function execution has resumed normal function for 30+ minutes. The incident will be marked as resolved. A post mortem will be conducted with the team tomorrow after further investigation tonight.
The issue has been pinpointed to our sharded function state database and a fix has been applied at ~ 2024-09-26T02:10:00Z UTC and errors have stopped. We are monitoring as we continue a deeper investigation.
Posted Sep 26, 2024 - 02:17 UTC
Identified
We've narrowed the issue down to a system component and are working on a resolution.
Posted Sep 26, 2024 - 02:07 UTC
Investigating
We're seeing a spike in functions that are failing to execute. We have been investigating and will update here accordingly.