Incident Summary
Our backend API response became unstable, rendering the Console and CLI unavailable for users. This instability disrupted services and impacted user experience.
What Happened
Our backend API experienced a combination of performance issues, including API gateway instability, Fastly performance degradation, increased incoming requests, and backend performance issues. These factors culminated in the disruption of API services, leading to the unavailability of the Console and CLI for users.
Customer Impact
The incident resulted in service disruptions for our users, preventing them from accessing critical functionalities through the Console and CLI.
Incident Resolution Steps
To address the incident and restore stability to the API, the following steps were taken:
Identified Root Cause and Impacts
The root cause of the incident is multifaceted, involving various components of our infrastructure:
These factors combined to create instability within the API, resulting in service disruptions and user inconvenience.