Incident has been resolved.

Next Steps

• Implement stricter rate-limiting and queue pacing for background jobs during peak hours.
• Audit ClickHouse resource allocation and concurrency limits to safely isolate background processing from user-facing workload traffic.

Impact

Intermittent delays and timeouts across all product surfaces

Root Cause:

Traffic spikes from background operations overloading ClickHouse

Incident Summary

Spotflow experienced periods of degraded performance and intermittent timeouts today across the entire platform. The root cause was identified as multiple sharp spikes in background operations traffic, which overloaded our primary ClickHouse instance and saturated database query resources.

Key Impacts & Mitigations

• Product Surfaces: Front-end dashboards and analytics queries experienced elevated response latency and intermittent execution timeouts.
• Background Queues: Non-essential asynchronous background operations were temporarily throttled to relieve pressure on the ClickHouse cluster.
• Database Recovery: Query execution times are returned to baseline levels as background traffic normalizes.