The networking fix has been applied and all systems are operational.
We have identified the root cause. Function execution has returned to normal. Any backlogs incurred during the incident will be executed.

The incident is now resolved and the system is full operational after monitoring.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

The incident is now resolved and the system is full operational.

After an extended monitoring period, function execution has returned to normal rates across all queue shards. During this issue, only a subset of users were affected on part of our infrastructure. Our infrastructure team is in the midst of rolling out additional system capacity going forward.

Runs, traces and events data are all caught up from their temporary backlog. The dashboard metrics are all being processed with no backlog.

System latency for function execution has returned to normal levels for the affected users. The incident has been resolved.

The cause of the incident was due to increased load causing congestion. We applied changes to the system to reduce congestion, resulting in increasing throughput. We also re-distributed some affected users in an effort to mitigate impact.

Our team's planned to roll out new infrastructure in the coming weeks and is accelerating that plan to aim to roll it out later this week to increase overall capacity.

We shipped a fix earlier today at 20:40 UTC (Jan 27) that has resolved the issue after an extended monitoring window to confirm the issue did not return.

The bug itself affecting some user requests was an issue due to a large "baggage" header which was beyond the limit of what the Event API's pubsub event stream can handle. Baggage headers are used for sending extra context within a request like open telemetry or APM tracing data. Some requests contained more than 1024 bytes which caused this issue. The fix applied earlier today gracefully now gracefully handles the situation where there are large baggage headers. This issue will not surface again.

Between 2:28 UTC and 2:41 UTC crons temporary failed to start. During this time window some functions also experienced delays.

Earlier today our run trace data pipeline in one of our regional clusters experienced some issues with writing data. This pipeline asynchronously ingests run traces and statuses into our database. Runs were executing successfully, but the traces could not be ingested, resulting in outdated run statuses in the Inngest dashboard (stuck in 'queued'). These runs continued to be executed, but the dashboard observability is out of date and in some cases, incorrect.

The system is fully operational now.

We are analyzing our system's data to determine the most effective way to prevent this case in all parts of our system that rely on this and similar data pipelines.

The fix has been running stable for 30 minutes and all system metrics are healthy. Run traces and metrics ingestion should be near realtime again.

The incident is now resolved and the system is full operational.

During this incident the system experienced a backlog in scheduling new runs and event data ingestion for the Inngest dashboard was delayed.

The root cause was due to an issue with our internal event streams brokers which power internal event ingestion, run trace data ingestion and metrics ingestion. This caused delays in other data in the dashboard as well.

The incident is now resolved and the system is full operational.

The incident is now resolved. The Inngest dashboard was down between 15:29 and 15:33 UTC. During this period the Inngest APIs were affected as well to load data and event keys which are required for the Event API to receive events.

The cause was an issue related to replication between databases. We are continuing to work with our database providers to understand the full impact and prevent the root cause from reoccuring.

Function execution was partially delayed for a subset of some users, but did not have an outage.

The incident is now resolved and the system is full operational. The issue started at 16:56:16 UTC and ended at 16:57:31 UTC.

The Event API caches event keys for a duration so not all requests would have been effected with sending events, but some may have. Our SDKs automatically retry up to 5 times with backoff, but due to the time window being >60s, many retries might have failed as well.

Our REST API and dashboards returned once the connections were re-established.

The cause of this incident is related to a networking issue that we are working on follow up mitigation efforts to avoid this issue from happening again.

The incident is now resolved and the dashboard is full operational.

After 18 hours of monitoring, the fix deployed yesterday resolved the incident.

System backlogs should be caught up and the system overall is stable with increased capacity. The root cause was determined and several system performance and reliability improvements will be added this week as follow ups.