Resolved - The issue was called by an erroneous configuration update in our CDN layer. This caused a massive increase in requests to our processing cluster which failed to scale fast enough due to unusual circumstances. After the issue was discovered, we initiated manual scale-up as well as rollback of the CDN configuration update. The incident started at 14:42 UTC and until 15:00 caused increased request latency and a significant amount of errors, By 15:00 UTC the errors were contained, and by 15:14 UTC, the system latency returned back to normal.
Sep 9, 13:07 EDT