Resolved
PubSub has been stable for hours after updates from the GCP team. The incident has been resolved.
Monitoring
Google Cloud has notified us and shared publicly some details about the Pub/Sub outage. We are continuing to work with their team to understand the scope of the outage and resolutions.
https://status.cloud.google.com/incidents/ghMho2Gka33Exr9UNavz
The Inngest Event API is has now been fully operational for 48 minutes (15:33 UTC) without another issue.
Identified
Event API and the rest of the Inngest system is fully functioning again as of 15:33 UTC. Issues began at 15:14 UTC. We are actively working with GCP's team to determine the root cause of the issues.
Identified
We've identified issues with Google Cloud PubSub, which we use to power event ingestion globally. This looks like a downstream provider issue. We are working to get in touch with GCP to determine resolution.
Investigating
Event API is coming back online intermittently. Investigating in ongoing.
Investigating
We are investigating an issue with event ingestion with our internal pub sub stream.