An alert channel with three hundred messages a day is the same as no alert channel. Technicians stop reading it, and the one message that mattered, the backup that has failed four nights running, scrolls past with the disk-space warnings from a server that has been at 85 percent for two years.
Good monitoring is mostly the discipline of turning things off. This post covers which checks earn a page, how to set thresholds that reflect real risk, and the routing that gets the right alert to the right person at the right hour.
Decide what an alert is for
An alert should mean a human needs to act now or soon. Everything else is a metric, a report, or a ticket that can wait until morning. Sort every check your RMM or monitoring tool offers into those buckets before enabling any of it.
The action test is useful: for each alert, write the action a technician should take when it fires. If the answer is 'look at it and probably do nothing', it is not an alert. If the answer is 'open a ticket for next week', it is a scheduled report. Only the checks with an immediate action survive as alerts.
- Page now: site offline, server down, backup failed, security product disabled, disk about to fill on a production volume
- Ticket automatically, work in hours: service stopped and did not restart, certificate expiring within two weeks, patch failed three times
- Daily digest: devices offline more than a day, pending reboots, agent version drift
- Monthly report: disk growth trends, warranty expiries, licence counts
Thresholds that reflect risk, not defaults
Vendor default thresholds are guesses. A disk at 90 percent on a 200 GB volume that grows a gigabyte a month is not urgent. A disk at 70 percent on a database volume that grows ten gigabytes a day is. Set thresholds per role, and where the tool supports it, alert on time-to-full rather than percentage.
CPU and memory alerts are the worst offenders. A server that sits at high CPU during a nightly job is healthy. Alert on sustained load over a longer window and only where it maps to user impact. For workstations, drop CPU and memory alerts entirely and rely on the helpdesk; nobody is going to act on a laptop being busy.
- Group devices by role: domain controller, file server, database, hypervisor, workstation, network
- Set disk thresholds per role, with a longer evaluation window for slow-growing volumes
- Replace CPU and memory alerts on servers with sustained-load checks over a long window
- Remove performance alerts from workstations entirely
- Review the top ten noisiest alerts monthly and fix or delete each one
Get-PSDrive -PSProvider FileSystem | Select-Object Name, @{n='UsedGB';e={[math]::Round($_.Used/1GB,1)}}, @{n='FreeGB';e={[math]::Round($_.Free/1GB,1)}}Dependencies and flapping
When a site's internet drops, every device behind it goes offline at once. Without dependency awareness that is forty alerts for one cause. Configure the monitoring tool so device checks depend on the site's gateway check, and suppress the children when the parent fails. Most RMM and network monitoring platforms support this; it is rarely turned on.
Flapping is the other noise source: a service that stops and starts every few minutes, an access point that reboots hourly. Add a minimum duration before an alert fires and a cool-down before it can fire again. The underlying fault still needs fixing, but it needs one ticket, not sixty pages.
Routing, hours and the on-call rule
Route by severity and time. Page-now alerts go to the on-call technician at any hour. Ticket alerts land in the queue with the right priority and nobody's phone buzzes. Digests go to the account owner by email in the morning. If a page fires and nobody acknowledges it within a set number of minutes, it escalates to the next person; write that chain down.
Then hold to one rule: every page gets a response, and every false page gets a fix. When a technician is woken up for something that did not need action, the next morning's work is to change the threshold, add the dependency, or delete the check. Treat noise as a bug in the monitoring system, because it is one.
- Page-now: on-call phone, escalates after a set number of minutes without acknowledgement
- Ticket: helpdesk queue with priority set by the alert, no notification outside hours
- Digest: email to the account owner each morning
- Every false page results in a change to the check the next business day
- Monthly: count pages per client, list the top causes, fix the top three
Frequently asked questions
Should clients receive alerts directly?
Not raw alerts. Send them a short daily or weekly summary, and a personal message when something material fails and what you are doing about it. Forwarding the alert stream makes them anxious and teaches them to ignore you.
How many alerts per day is reasonable?
For a well-tuned environment, pages should be rare and each one should be real. If a technician sees more than a handful of pages a day across all clients, something is misconfigured.
What should be monitored on network devices?
Reachability of the gateway, WAN link state and errors, interface errors on uplinks, and configuration changes. Skip bandwidth alerts unless a link saturating is an actual problem for that site.
Takeaway
Monitoring that works is a short list of checks that each map to an action, thresholds set per role, dependencies so one outage is one alert, and a routing chain that reaches a person at the right hour. Then treat every false page as a bug and fix it the next day. The channel goes quiet, and when it speaks, people move.