This incident has been resolved.
Feb 1, 13:02 PST
We have removed the last mitigation measure in place and are seeing API latency and error rate back to normal levels. We will continue to monitor.
Feb 1, 12:35 PST
We are seeing elevated API latency and error rate again after removing the mitigation measures in place. We are removing them one at a time and monitoring closely.
Feb 1, 11:58 PST
We are seeing API latency and error rate back to normal. We will now proceed with reverting some mitigation measures still in place and keep you posted on how that goes.
Feb 1, 11:30 PST
The rollback is complete. We are monitoring.
Feb 1, 10:59 PST
Our rollback is almost complete and we are seeing a positive trend in the latency and error rate.
Feb 1, 10:52 PST
We are halfway through rolling back our previous deployment. We will keep you posted when it's done. Thank you for your continued patience.
Feb 1, 10:10 PST
We are currently rolling back a change and hope to see positive results shortly.
Feb 1, 09:17 PST
We are reverting the previous deployment that we suspect is related to this issue.
Feb 1, 08:57 PST
We are continuing to investigate the source of the the problem.
Feb 1, 08:57 PST
We are seeing an improvement in response times, error rates are dropping.
Feb 1, 08:20 PST
We believe we have determined the cause are are looking to block the source of the traffic.
Feb 1, 08:00 PST
We are continuing to see an increase in 499 and 502 responses and we are still investigating the root cause.
Feb 1, 07:30 PST
We are still receiving reports about issues with API endpoints, we are investigating the issue.
Feb 1, 06:46 PST
Services are back to normal now. We are monitoring the endpoints.
Feb 1, 06:22 PST
A fix has been implemented and the services are coming back to normal.
Feb 1, 06:21 PST
We are investigating this issue at the moment. Will provide further updates here.
Feb 1, 06:19 PST