Heartbeat went down although the job clearly ran

Question

Our nightly job pinged the check-in URL on time, I can see the request in our own logs. MonoDuty still marked the heartbeat as down and opened an incident at the same minute the ping arrived. The incident description mentions something about the body not matching.

Deniz Arslan · Accepted Answer

That explains it exactly, an empty body cannot contain DONE. You have two clean options:

Option one, drop the requirement. Edit the heartbeat and clear the expected body field. From then on any ping counts as a success and only lateness can bring the heartbeat down. This is what most teams want.

Option two, keep it and send the body. This is genuinely useful when you want the job to be able to report failure explicitly. Have your script send the text in the request body, for example by posting DONE only on the success path. Then a job that finishes but detects a problem can ping with different content and MonoDuty will treat it as failed even though a request arrived.

Whichever you choose, resolve the currently open incident afterwards. Note that once the next successful ping arrives and the heartbeat transitions from down back to up, linked open incidents are auto-resolved for you.

Deniz Arslan · Answer

Hi Daniel, that description points at the expected body setting. If a heartbeat has an expected body configured, the raw request body of the ping must contain that substring. A ping that arrives perfectly on time but with the wrong body is recorded as failed and flips the heartbeat down. Do you have a value in that field?

Daniel Okoye · Answer

Yes, someone on the team set it to DONE last month. Our job pings with an empty body from curl.

Daniel Okoye · Answer

We went with option two and now post DONE explicitly. Nice that it doubles as a failure signal.

Heartbeat went down although the job clearly ran

3 replies