Resolved -
This incident has been resolved.
Feb 1, 13:02 PST
Update -
We have removed the last mitigation measure in place and are seeing API latency and error rate back to normal levels. We will continue to monitor.
Feb 1, 12:35 PST
Update -
We are seeing elevated API latency and error rate again after removing the mitigation measures in place. We are removing them one at a time and monitoring closely.
Feb 1, 11:58 PST
Update -
We are seeing API latency and error rate back to normal. We will now proceed with reverting some mitigation measures still in place and keep you posted on how that goes.
Feb 1, 11:30 PST
Monitoring -
The rollback is complete. We are monitoring.
Feb 1, 10:59 PST
Update -
Our rollback is almost complete and we are seeing a positive trend in the latency and error rate.
Feb 1, 10:52 PST
Update -
We are halfway through rolling back our previous deployment. We will keep you posted when it's done. Thank you for your continued patience.
Feb 1, 10:10 PST
Identified -
We are currently rolling back a change and hope to see positive results shortly.
Feb 1, 09:17 PST
Update -
We are reverting the previous deployment that we suspect is related to this issue.
Feb 1, 08:57 PST
Update -
We are continuing to investigate the source of the the problem.
Feb 1, 08:57 PST
Update -
We are seeing an improvement in response times, error rates are dropping.
Feb 1, 08:20 PST
Update -
We believe we have determined the cause are are looking to block the source of the traffic.
Feb 1, 08:00 PST
Update -
We are continuing to see an increase in 499 and 502 responses and we are still investigating the root cause.
Feb 1, 07:30 PST
Investigating -
We are still receiving reports about issues with API endpoints, we are investigating the issue.
Feb 1, 06:46 PST
Monitoring -
Services are back to normal now. We are monitoring the endpoints.
Feb 1, 06:22 PST
Identified -
A fix has been implemented and the services are coming back to normal.
Feb 1, 06:21 PST
Investigating -
We are investigating this issue at the moment. Will provide further updates here.
Feb 1, 06:19 PST