Back to overview
Downtime

Swaps API is unavailable

Sep 02 at 08:20am EDT
Affected services
MACE Swaps API
MACE Swaps API
Morgenstern - Umbrasync
Umbrasync -> Block number
Umbrasync -> Blocks behind

Resolved
Sep 02 at 10:02am EDT

Production swaps servers were restarted manually, beta servers recovered automatically.

Updated
Sep 02 at 08:55am EDT

Umbrasync database was cleared manually and measures will be taken so that automated recovery is less likely to fail in the future. The swaps API should recovery automatically in the next 5 to 10 minutes, if not, they will be restarted manually.

Updated
Sep 02 at 08:49am EDT

It is now known why the automated recovery measures failed. The recovery process involved clearing the chain snapshot, which took more than the timeout time given.

Created
Sep 02 at 08:20am EDT

At approximately 1AM EDT, Umbrasync started having a noticeable delay in providing real-time data to our swaps solver. Automated recovery measures were attempted at 4AM EDT which failed. The issue is currently being diagnosed