Creating granular alert notifications – rule by rule, monitor by monitor

There are a couple resources on the web about how to do this from C# and Powershell.  The best sites I have seen are at:

http://blogs.msdn.com/jakuboleksy/archive/2007/01/18/notification-subscriptions.aspx

and

http://code4ward.net/cs2/blogs/code4ward/archive/2007/09/19/set-notificationforalert.aspx

These examples are cool – showing us we can create more granular criteria than the choices exposed in the UI.

In the second link above – the basic Powershell script is posted.  However, many users have been looking for examples of how to add more specific and typical criteria.  The script above will notify on any change in resolution state…. so if you use multiple resolution states, it might send an email when the alert is created, then modified, then closed.  Many want a notification only when an alert comes in New.

The posted script also shows the Last Modified time in the notification in GMT…. where as a UI generated subscription would show the alert time in the correct local time zone. 

The post below will show some of the changes you can make here, and some example scripts.

First things first…. let’s talk about what happens when you create a normal subscription in the UI:  When you run the New Subscription wizard in the UI, you are selecting from what was determined to be the most typical choices for subscribing to alerts for a notification.  What this does, behind the scenes, is create some XML, and writes that information to the “Notifications Internal Library” management pack (Microsoft.SystemCenter.Notifications.Internal)

The best thing you can do to understand what is happening – is to create a SINGLE subscription in the UI, and then export this management pack, and open it in Notepad.  What you will see – is that this MP holds all the configuration information for your SMTP server settings, default notification format, all the recipients and their information, product connector subscriptions, and the focus of this discussion: notification subscription details.

Find your simple subscription that you created – and look at the options.  This will start with:

<Monitoring>
    <Rules>
      <Rule ID=”Subscription(guid) plus other settings……..>
        <Category>Notification</Category>

Look at all the default options…. this will help you understand the XML format and settings for notifications.  Probably the most important to understand is <Criteria>.  What you will see is all the criteria settings you just created, using the UI.  Unfortunately – the UI does not expose all the possible settings for criteria….

So – you may ask – what settings are available?

Well – according to Jakub:  “In terms of which columns you can query for in the criteria, it is anything that is defined on the Alert table in the db.”

Whoa!  That’s a LOT more options.  The following columns exist in the alert table:

AlertId   
AlertName  
AlertDescription  
BaseManagedEntityId  
ProblemId  
IsMonitorAlert  
RuleId  
ResolutionState  
Priority  
Severity  
Category  
Owner  
ResolvedBy  
TimeRaised  
TimeAdded  
LastModified  
LastModifiedBy  
LastModifiedExceptRepeatCount  
TimeResolved  
TimeResolutionStateLastModified  
TimeResolutionStateLastModifiedInDB  
CustomField1  
CustomField2  
CustomField3  
CustomField4  
CustomField5  
CustomField6  
CustomField7  
CustomField8  
CustomField9  
CustomField10  
TicketId  
Context  
RepeatCount  
AlertStringId  
AlertParams  
SiteName  
ConnectorId  
LastModifiedByNonConnector

So you can see – if you don’t mind a little custom XML work…. you have a lot more choices when it comes to notifications.  We can use this to create subscriptions to specific named alerts, or only alerts with a specific custom field value, a specific repeat count threshold, or for a specific computer, or resolution state.  The possibilities get pretty good.  If only this was all exposed in the UI!

Understanding, that once custom modified in XML – you will not be able to ever modify the same (non-standard) subscription in the UI, or you will break your XML customizations.

Ok – beyond criteria – what else is interesting, you ask?

<PollingIntervalMinutes>   This defaults to 1 minute and should not be changed.  This is how often the subscription will poll the SDK to look for new alerts and see if they match any notification subscriptions.

<TimeZone>  Using the script as posted – it did not contain this tag.  Therefore all alerts come in as GMT.  Using this tag – you can set your specific time zone.  Central standard time looks like this:

<TimeZone>6801000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000|Central Standard Time</TimeZone>

If you want to know your timezone code – just create any subscription in the UI – look at the Notifications Internal Library MP, and pull it out of there for use in scripting.

<IdleMinutes>  This is the tag to use for Alert aging.  This is used to delay a notification from being sent for “x” minutes, as long as it has not been changed in that time and meets all other criteria.  This is used to keep notification noise down, if alerts are closed or acknoledged quickly, or to create “nag” notifications in additional subscription to “re-notify” after “x” amount of time has passed – and nobody acknowledged or closed the alert.  NOTE:  In order to use this tag – the class targeted MUST be set to “AlertNotChangedSubscriptionConfiguration”…. instead of the default which is “AlertChangedSubscriptionConfiguration”.  The BEST way to understand this – is to create TWO IDENTICAL subscriptions to alerts, but set one to have alert aging enabled, set the other to not have it enabled.  What you will see is that alert aging subscriptions have a XML tag like the following:

<AlertNotChangedSubscription Property=”Any”> </AlertNotChangedSubscription>

Where standard subscriptions that do not utilize alert aging will have the following:

<AlertChangedSubscription Property=”Any”> </AlertChangedSubscription>

Make sure you keep these clear.  More on using these in scripts later.

Ok – that covers the most notable XML items.  I would say, it is probably easiest to just configure the XML directly when you want to use these advanced properties that are not exposed in the UI.  However, if you have the need/desire to create these types of subscriptions from script… we can do that, and let’s see a couple examples.

Using the Powershell code posted from http://code4ward.net/cs2/blogs/code4ward/archive/2007/09/19/set-notificationforalert.aspx we want to make a couple changes right off the bat.

First thing – is we wont use the ProblemID field as in that example.  This field of an alert is only consistent on Monitors.  However – for any alert generated by a Monitor – the SDK property of  ProblemID will be identical to MonitoringRuleID.  Therefore, we can always use MonitoringRuleID property which will be unique and consistent, for a given alert, whether it came from a rule or a monitor.

To see these values – use get-alert in Powershell.  For a more specific command to see just these properties on new alerts:  get-alert | where {$_.ResolutionState -eq 0} | ft name,problemid,monitoringruleid.

By the way – in the database, the SDK property “ProblemID” = the DB column “ProblemID”.  The SDK property “MonitoringRuleID” = the DB column “RuleID”.

Ok – so we will make the changes to use MonitoringRuleID and RuleID for all alerts.  You can see the example in my attached script below… and there is an example posted at the link above.

Next – we will add the timezone information…. this is done by adding a new line under “$config.PollingIntervalMinutes = 1”.  The new line should look like (without wrapping, and this example is for CST):

$config.TimeZone = “6801000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000|Central Standard Time”

Lastly – we will modify the Alert Criteria with our custom subscription criteria.  What we want to do – is subscribe to any alert with a given “MonitoringRuleID” GUID, which using this script, will query that value from the common Alert Name we see in the console.  We can use any criteria here.  Things to keep in mind – the first <expression></expression> tags are not necessary, as they are understood.  Any other AND/OR groupings have to be done manually.  Here is the code from the sample script (this should always be one line in the script – I wrapped it here to make it readable):

$config.Criteria =
“<SimpleExpression>
   <ValueExpression>
     <Property>RuleId</Property>
   </ValueExpression>
     <Operator>Equal</Operator>
   <ValueExpression>
     <Value>$MonitoringRuleId</Value>
   </ValueExpression>
</SimpleExpression>”

What the above will do – is to create a subscription that will match on the specific alert name given to the script, and then create a notification on ANY change to the alert…. such as when the alert is generated (New) AND when the alert is closed.

Let’s add some new criteria using an AND statement – which will ONLY subscribe to NEW alerts:

$config.Criteria =
“<And>
  <Expression>
    <SimpleExpression>
      <ValueExpression>
        <Property>RuleId</Property>
      </ValueExpression>
        <Operator>Equal</Operator>
      <ValueExpression>
         <Value>$MonitoringRuleId</Value>
      </ValueExpression>
    </SimpleExpression>
  </Expression>
  <Expression>
    <SimpleExpression>
      <ValueExpression>
        <Property>ResolutionState</Property>
      </ValueExpression>
        <Operator>Equal</Operator>
      <ValueExpression>
        <Value>0</Value>
      </ValueExpression>
    </SimpleExpression>
  </Expression>
</And>”

That will work MUCH better…. if we only want to see a notification for new alerts.  Keep adding to the criteria sections to get the subscription to match exactly what you want.

The only other things I changed in the script – was the echoed statement about what was shown to the user running the script:

write-Host “Executing script to subscribe to specific alert by name…”

And then I changed the formula for naming the subscription.  Since these CANNOT be modified in the UI without breaking them – I changed the default naming for these script generated subscriptions:

$Subscription.DisplayName = “Script created – Do NOT MODIFY in UI – ” + $NotificationRecipientName + ” – ” + $AlertName

This creates subscriptions in the GUI that tip any admin’s off – to leave them alone:

Lastly – if we want to use Alert aging in our script – we need to change a couple of items.  First – we need to add a new line for $config.IdleMinutes under the “$config.PollingIntervalMinutes = 1”.  Here is an example:

$config.IdleMinutes = 60

That will add an hour delay – and only notify when the alert has been unchanged, and in the console for an hour.  In order to use alert aging delay – we must also change the “$config = New-Object” line from “AlertChangedSubscriptionConfiguration” to “AlertNotChangedSubscriptionConfiguration”.  Therefore – in order to use alert aging – you need a different script, than if you dont want alert aging…. and if you have different criteria – you will need a script for each scenario. 

Below – I am attaching a zip file with a couple script examples…..

One script to subscribe to NEW alerts without alert aging, and another with alert aging.

Tips:  It is probably easier just to edit the Notification MP XML for your customizations.  Just remember – to ALWAYS back this file up…. and keep several copies, in case you ever modify the file in a way that is not supported.