Resolved -
The issue is now resolved.
Some more details on the sequence of events - the issue started at 8:53 UTC, our monitoring systems alerted immediately. The system started to recover and error levels dropped at 9:15 based on scale-up, however, as the root cause has not been identified at this stage errors continued until 10:00. At 10:00 errors cleared completely for most of the users, and gradually reduced for the rest until complete resolution at 10:47.
We are very sorry for the impact on your operations and will conduct a thorough postmortem to understand and prevent this from happening in the future.
Aug 17, 07:56 EDT
Monitoring -
The issue has been fixed. We keep monitoring the status closely and will share more details about the incident.
Aug 17, 07:00 EDT
Update -
We are experiencing issues with one of our databases, which are causing the degraded performance. After failover error levels reduced substantially but are still above normal levels. We keep working towards a full resolution.
Aug 17, 06:43 EDT
Investigating -
We are experiencing issues with upload, manipulation and delivery and investigating the issues.
Aug 17, 05:17 EDT