The Uptime Monitor That Was Down for 27 Days

This week a routine task failed. The platform tried to set up monitoring for a new client site and got a 502 Bad Gateway when it logged in to Checkmate. I assumed a blip and retried. Same error. When I looked, the monitor's web app was crash-looping because it couldn't reach its database, and the database container had been stopped since late September.

The monitor had been down for 27 days. Nothing told me, because the thing that tells me when things are down was the thing that was down.

How a cleanup deleted the data

I rebuilt the timeline from logs of an earlier agent session. A month ago the server's disk filled to 100% while two large builds ran at the same time. The MongoDB container stopped around then. Its own logs are gone, so I'm inferring the cause, but a database that can't write to a full disk is the obvious suspect.

To free space, the agent I was working with changed Docker cleanup from daily to hourly. Then, with my approval, it ran a manual cleanup with delete unused volumes turned on. That reclaimed about 2 GB of build cache and about 550 MB of volumes. The runbook was then updated to say unused-volume cleanup should stay on.

When I restarted the database this week, Checkmate's health endpoint came back green within a minute. Its data didn't: zero users, zero monitors, no history. The volume was gone, and that cleanup is almost certainly what removed it.

The lesson is about one word. To Docker, a volume is "unused" when no container currently references it. It doesn't mean nobody needs it. A stateful database that's stopped, or whose container has been recreated, can look exactly like leftover junk from an old build. "Delete unused volumes" is a fine setting on a box that only runs stateless apps. On a box that hosts databases, it's a slow-motion data loss bug waiting for the wrong moment.

I approved that cleanup myself. The agent explained what it was going to do, and it all sounded reasonable for a full disk. Nobody asked which volumes counted as unused, and that question was the one that mattered.

The second bug: trusting a stored ID

Recovery turned up a quieter problem in my own code. When the platform creates a Checkmate monitor, it stores the monitor's ID. On later runs, if a stored ID exists, it assumes the monitor exists and skips creation.

That's a normal optimization, and here it meant the platform would never fix itself. Every site still had an ID pointing at a monitor that no longer existed. Bring Checkmate back empty, and the platform would still report everything as provisioned.

The fix was to verify before trusting:

• Before relying on a stored ID, ask the provider whether that monitor still exists.
• Only recreate it when the answer is a confirmed "not found", cross-checked against the full monitor list. A single 404 isn't enough.
• If the provider is unreachable or returns a server error, never recreate anything. Fail visibly and try later.

That last rule matters. When "I can't reach it" and "it doesn't exist" go down the same path, a network hiccup turns into duplicate monitors. A recreated monitor reuses the same database row, so there's still one record per site, and each relink is logged as an event I can audit later.

The third bug: logging in on every call

After I recreated the Checkmate admin account, the relink sweep failed halfway through with 429 Too Many Requests. The provider client logged in fresh on every API call, about three logins per check, and one sweep fired roughly 20 logins in two seconds. Checkmate rate-limits logins, which it should.

The safety design held. Failed relinks stayed marked as failed, nothing got duplicated, and a second run finished the job. A confirming run found nothing missing. The real fix was simple: cache one login token per process and reuse it until the server rejects it. I'd never seen the problem before because the code had only ever handled one or two sites at a time.

This is the pattern I keep hitting with integrations: code written for the single-item path ends up running in a loop during recovery, at the worst possible moment.

What changed on the server

The data was gone, so the work was about making sure it can't happen quietly again:

• Unused-volume deletion is off. Build cache and old images still get cleaned on a schedule. Volumes don't.
• One build at a time. Two builds at once is what filled the disk. Builds are a bit slower now, and the server never runs out of space.
• Container status alerts are on. Coolify can email when a container changes state. A database stopping is now an email, not a mystery I discover a month later.
• Every database has scheduled backups, including the monitor's. Each backup job was test-run, not just configured. Getting copies off the same host is the next step.
• The runbook was corrected. The old line telling future agent sessions to keep volume cleanup on is gone. A wrong instruction in a runbook keeps causing damage long after the incident.

Watch the watcher

The uncomfortable part isn't the deleted volume. It's the 27 days. My monitoring was a single point of failure with no monitoring of its own, and its failure mode was silence. Silence looks exactly like everything being fine.

A few things I'd tell anyone running their own monitoring stack:

1. Your monitor needs an outside check. Something that doesn't share its host, its database or its alert channel should confirm the monitor is up and actually running checks. A free external ping on the monitor's own health URL is enough to start.
2. Alert on state changes, not just on outages. "Database container stopped" is boring, and it's the alert that would have saved a month here.
3. Read cleanup settings as deletion policies. Before turning on anything that removes "unused" things, write down what it'll consider unused on this host. If the list could include a stopped database, turn it off.
4. Don't let stored IDs from another system become facts. Check them now and then. Treat "not found" and "unreachable" as different answers.

The monitor is back, every site has a check again, and I now know about a stopped container within minutes instead of weeks. What I lost was a month of uptime history. That's a cheap price for the lesson, and I'd rather pay it on my own monitoring than on a client's data.