We shipped a fix earlier today at 20:40 UTC (Jan 27) that has resolved the issue after an extended monitoring window to confirm the issue did not return.
The bug itself affecting some user requests was an issue due to a large "baggage" header which was beyond the limit of what the Event API's pubsub event stream can handle. Baggage headers are used for sending extra context within a request like open telemetry or APM tracing data. Some requests contained more than 1024 bytes which caused this issue. The fix applied earlier today gracefully now gracefully handles the situation where there are large baggage headers. This issue will not surface again.
Resolved
We shipped a fix earlier today at 20:40 UTC (Jan 27) that has resolved the issue after an extended monitoring window to confirm the issue did not return.
The bug itself affecting some user requests was an issue due to a large "baggage" header which was beyond the limit of what the Event API's pubsub event stream can handle. Baggage headers are used for sending extra context within a request like open telemetry or APM tracing data. Some requests contained more than 1024 bytes which caused this issue. The fix applied earlier today gracefully now gracefully handles the situation where there are large baggage headers. This issue will not surface again.
Identified
We're seeing an increased number of errors reported with users publishing events with our internal pubsub due to additional metadata. The team is actively working to mitigate this error.