Why false positives are dangerous
A monitor that cries wolf is worse than no monitor, because it trains your team to ignore it. After enough 3 a.m. pages that turn out to be nothing, people mute channels, add filters, and stop trusting alerts — and then miss the real outage buried in the noise. Reducing false positives is not about comfort; it is about keeping the signal credible so that a page still means "act now".
Confirm before you alert
Most transient failures resolve themselves within seconds: a dropped packet, a brief connection reset, a garbage-collection pause. Requiring several consecutive failures before opening an incident filters those out while still catching real outages quickly. PageLantern opens an incident after two consecutive failed checks by default and resolves it on the first successful check, and you can raise either threshold for flaky targets. If a target flaps, raising the recovery threshold above its default of one makes the service confirm the recovery before it closes the incident.
Check from more than one place
A single vantage point cannot tell "the service is down" apart from "the path between one prober and the service is down". A blip in one network should not page anyone if every other location still sees the service as healthy. Running checks from multiple regions and treating a failure as real only when several agree removes a whole category of false alarms caused by transient network issues near the monitor rather than the service.
Set realistic timeouts and intervals
A timeout that is tighter than your service's real worst-case response time manufactures failures out of ordinary slow responses. Set it above your true p99, not at the average. Match the check interval to how critical the target is: over-frequent checks on a naturally variable endpoint generate more blips to alert on without detecting real outages any sooner.
Alert on symptoms, and use maintenance windows
Alert on what users experience — the request is failing, the page is down, the job did not run — rather than on every internal cause. Symptom-based alerts are both fewer and more actionable, and they do not fire for a condition that has no user impact. Finally, suppress alerts during planned work with a maintenance window, so a deliberate deploy or migration does not page the on-call engineer who is doing it.
Treat every false positive as a small bug: when one slips through, tune the threshold, timeout, or scope that let it. Over time that discipline is what keeps alerts worth waking up for.