I’ve been super busy with a new job, but came across this situation and decided a quick blog post was in order.
Several SCOM alerts that had been generated by monitors were still open but the monitor that generated them had returned to a healthy state. This was confusing for the event management team. They rightly expected the monitors to close the alerts once the error condition was resolved.
What was going on?
Looking at the History tab of the alert, showed the alert had been resolved by the system then modified by a user.
The event management team don’t see resolved/closed alerts in the views they have, how did they update a closed alert setting it’s resolution state to ‘Acknowledged’’? When you multi-select alerts the Operations Consoles pauses refreshes until you select a single alert or hit refresh.
The event monitoring team operator was responding to the heartbeat failure alerts and had selected the alerts in the console to change the resolution state to ‘Acknowledged’. Whilst this was happening the monitor returned to a healthy state and resolved the alert. Because refresh had been disabled the operators view did not update with the new status meaning the alert was effectively reopened by the operator.
Because of this we can’t be sure that the open alerts in the console should still be open. Fortunately we can use an Operations Manager PowerShell ‘one-liner’ to close the alerts in this state.
Get-ScomAlert -criteria ‘ResolutionState<”255” AND IsMonitorAlert=”True” AND MonitoringObjectHealthState=”1”’ | Set-SCOMAlert -ResolutionState 255