Pod Health Alerts End When Your Container Recovers
A restart or out-of-memory alert now ends once the container has settled, and one incident sends one notification instead of several.
What's changed
- A restart or out-of-memory alert on a database, cache, queue, managed app, VPS or Rapids container now ends by itself once the container has run for 30 minutes without another restart (longer, up to 2 hours, when its run before the last restart was itself long); before, it could stay open for as long as the container lived.
- The High restart count alert now needs 5 or more restarts within the last 2 hours, instead of 5 restarts at any time since the container was created, so an old, quiet container no longer raises it.
- The banner on a database, cache, queue or managed app page now says how many times the container restarted in the last 2 hours and when it last restarted, and the High restart count email gives the same count, instead of a lifetime count with a Last seen time that moved on every check.
- One incident now sends one notification: out of memory, crash loop and restart count on the same container are announced once, a replica failing alongside the first or the container that replaces a failing one is not announced again, an alert our team resolved by hand does not notify you a second time for the same problem, and a container killed for memory again within 6 hours of the last such alert is not announced again; a problem that gets worse, or a different problem, is still announced.
- Alert emails and in-app notifications are now sent from a queue, one team member at a time, with up to two retries, so one address the mail provider refuses no longer stops the rest of your team from being told or cuts short the health check of your other resources.
- A notification still waiting to be sent when its alert has ended or been handled by our team is dropped instead of arriving late.