For 18 minutes, between 23:40 and 23:58 UTC, customers have experienced increased latencies and error rates when uploading and generating new images. This global slowdown was caused by a singularly anomalous media resource, whose processing created an overflow. This resource was used by a popular website. This combination resulted in a non-malicious DDOS attack on our servers. Our monitoring systems notified us immediately of the situation, and our incident analysis systems helped us isolate the offending resource and we eventually blocked it from our system. We are working to add protections beyond the ones that already have and failed, that will prevent such incidents from recurring.
Posted Apr 24, 2016 - 21:29 EDT
Monitoring
Error levels and latencies are back to normal, we are working to prevent recurrence of this event.
Posted Apr 24, 2016 - 20:24 EDT
Identified
We've found the requests causing the system's increased latencies, and have disabled them to mitigate their effect.
Posted Apr 24, 2016 - 20:00 EDT
Investigating
Since 23:40 UTC today we are seeing increased error level and latencies for new image generation. We are investigating the root cause.