All services have been fully restored. Affected Gmail accounts are now syncing properly.
Posted Apr 24, 2020 - 19:46 PDT
Update
Gmail accounts first connected between April 22 00:01 UTC and April 24 23:30 UTC may have incomplete historical mail data. We are working to ensure all of those accounts have complete historical mail data. These affected accounts are properly receiving new data. Sync is behaving normally for all other accounts.
Posted Apr 24, 2020 - 16:45 PDT
Update
Sync has fully recovered. API and webhooks remain healthy. We are monitoring the situation to ensure nothing degrades.
Posted Apr 24, 2020 - 16:33 PDT
Monitoring
API & webhooks continue to look healthy. Sync has mostly recovered. We are rebalancing our sync fleet to improve performance for a small percentage of sync accounts.
Posted Apr 24, 2020 - 14:08 PDT
Update
API & webhooks continue to look healthy. Sync recovery is still ongoing. A vast majority of accounts are syncing normally. We are still investigating some degraded account syncs.
Posted Apr 24, 2020 - 10:41 PDT
Update
API success rates have stabilized. We are still actively monitoring a hard-hit database node that, while stabilizing, appears to have been at the root of API success degradation over the past hour.
The webhook backlog has been substantially alleviated.
The syncing service is still steadily improving, but not yet at full health.
Posted Apr 24, 2020 - 09:28 PDT
Update
The sync fleet is steadily returning to normal; however, due to lots of backlogged activity, we are seeing intermittent degradation in API requests, as well as a higher than normal backlog of webhooks with increased latency. We are actively monitoring this backlog and expect gradual recovery . We are working on adding capacity to alleviate this sooner.
Posted Apr 24, 2020 - 08:40 PDT
Update
We are continuing to work on unpausing the sync fleet.
Posted Apr 24, 2020 - 07:40 PDT
Update
We are still experiencing issues and approximately 1/2 of the sync fleet is still paused.
Posted Apr 24, 2020 - 04:56 PDT
Update
We have unpaused a very small portion of the fleet that had previously been paused, but are waiting for improvement in the platform health before proceeding with more.
Posted Apr 24, 2020 - 03:12 PDT
Update
We are starting to unpause our sync fleet.
Posted Apr 24, 2020 - 02:21 PDT
Update
API success rates are back up at 99.9%. 1/2 of the sync fleet is still paused.
Posted Apr 24, 2020 - 01:39 PDT
Update
We have paused 1/2 our sync fleet.
Posted Apr 24, 2020 - 01:22 PDT
Update
We are pausing a portion of our sync fleet in order to stabilize our databases
Posted Apr 24, 2020 - 00:26 PDT
Update
We are continuing to work on a fix.
Posted Apr 23, 2020 - 23:42 PDT
Update
The platform is starting to stabilize. API success rates are currently at 75%.
Posted Apr 23, 2020 - 22:49 PDT
Identified
We are seeking to stabilize the platform. API success rates are currently 80%.
Posted Apr 23, 2020 - 22:12 PDT
Investigating
We are currently investigating a degradation of API success rates. API success rates are currently at 90%
Posted Apr 23, 2020 - 21:24 PDT
This incident affected: Nylas Application Services (API, Sync Engine).