Earlier today our run trace data pipeline in one of our regional clusters experienced some issues with writing data. This pipeline asynchronously ingests run traces and statuses into our database. Runs were executing successfully, but the traces could not be ingested, resulting in outdated run statuses in the Inngest dashboard (stuck in 'queued'). These runs continued to be executed, but the dashboard observability is out of date and in some cases, incorrect.

The system is fully operational now.

We are analyzing our system's data to determine the most effective way to prevent this case in all parts of our system that rely on this and similar data pipelines.

The fix has been running stable for 30 minutes and all system metrics are healthy. Run traces and metrics ingestion should be near realtime again.

The incident is now resolved and the system is full operational.

During this incident the system experienced a backlog in scheduling new runs and event data ingestion for the Inngest dashboard was delayed.

The root cause was due to an issue with our internal event streams brokers which power internal event ingestion, run trace data ingestion and metrics ingestion. This caused delays in other data in the dashboard as well.

The incident is now resolved and the system is full operational.

The incident is now resolved. The Inngest dashboard was down between 15:29 and 15:33 UTC. During this period the Inngest APIs were affected as well to load data and event keys which are required for the Event API to receive events.

The cause was an issue related to replication between databases. We are continuing to work with our database providers to understand the full impact and prevent the root cause from reoccuring.

Function execution was partially delayed for a subset of some users, but did not have an outage.

The incident is now resolved and the system is full operational. The issue started at 16:56:16 UTC and ended at 16:57:31 UTC.

The Event API caches event keys for a duration so not all requests would have been effected with sending events, but some may have. Our SDKs automatically retry up to 5 times with backoff, but due to the time window being >60s, many retries might have failed as well.

Our REST API and dashboards returned once the connections were re-established.

The cause of this incident is related to a networking issue that we are working on follow up mitigation efforts to avoid this issue from happening again.