Monitors

How Often Should You Check? Choosing a Monitoring Interval That Makes Sense

MontyDec 25, 2024 5 min read

"Check every minute" is the default answer, and for a lot of services it is the right one. But it is worth understanding the trade you are making, because the interval interacts with retries, timeouts and alert noise in ways that surprise people.

The detection latency maths

If a service goes down at a random moment, your average detection delay is roughly half the interval, and your worst case is the full interval. But that is only the first term. Retries add to it. If your timeout is thirty seconds and the failure is a hang rather than a refusal, each attempt burns the full timeout.

Worst case detection is: interval + (attempts × timeout) + (gaps between attempts). With a one minute interval, a thirty second timeout and two retries at five second gaps, your worst case is 60 + 90 + 10 = 160 seconds. Nearly three minutes, from a setup that looks like it should detect in one.

What actually drives the choice

How fast can you respond anyway? If your on-call person takes ten minutes to get to a laptop, shaving detection below that is optimising the wrong term. What is the cost per minute of the outage? Match the interval to the consequence. How stable is the target? A flaky service produces more false positives at aggressive settings. What does the check cost the target? A deep health endpoint queried every ten seconds is a permanent load you added to production.

Reasonable defaults

Customer-facing critical paths: one minute with two or three retries. Important but non-critical: five minutes. Slow-moving external dependencies: ten to fifteen minutes. Anything scheduled rather than always-on: heartbeats, not interval checks.

The flapping problem

A target sitting right at the edge of your timeout oscillates between up and down, and each transition opens or closes an incident. Fix the target if you can — flapping usually means something is genuinely marginal. Otherwise raise the retry count, or raise the timeout only if you genuinely accept the slower response.

A worked example

A payments API. Interval one minute. Timeout three seconds, well above p99 and well below the point a user gives up. Retries two at five second gaps. Worst case detection: 60 + (3 × 3) + 10 = 79 seconds. Notice the timeout did most of the work — setting it near the user's tolerance rather than near TCP's patience is the highest-leverage decision in the whole configuration.

Monty