On April 4th at approximately 9:00PM EST, an automated billing script mistakenly froze approximately 50% of our customer applications, causing API calls using these application IDs to return "403 - This application has been disabled" errors. The issue was brought to our attention by customers at approximately 11:20 PM, and we immediately took action to unfreeze all accounts, completing the process by 12:00 AM. The root cause was an update to the script that did not properly account for changes in our billing system.

To prevent future incidents, we have improved our database to establish a "single source of truth" for paid status, added safety guards and monitoring to these scripts, implemented anomaly detection for all HTTP status codes (not just server errors), and set up a system to further monitor support ticket influx in a short period of time. We treat any situation where the customer notifies us of an incident before our systems alert us to the situation as a fundamental flaw, so we prioritize the remediation of these flaws at the highest level.

We apologize for any inconvenience caused and are committed to refining our systems and processes to provide the best possible service.