Setting up notifications for email, IM, or command channels is almost identical to how this was configured in OpsMgr 2007 R2. This article will just serve as a walk through to the process, such as immediately after deploying OpsMgr 2012. The key difference here is that Notifications are now managed by a Resource Pool, instead of just depending on the RMS.
Notifications in OpsMgr are made of of three primary components – the Channel, Subscriber, and the Subscription. The Channel is the mechanism that we want to notify by, such as Email. The subscriber is the person or distribution list we want to send to, and the subscription is a definition of criteria around what should be sent.
The SMTP Channel:
We will first need to create the channel: Under Administration pane > Notifications > Channels. Right click and choose New channel > Email (SMTP)
Give your channel a name. We might have multiple email channels. Once for emails to our primary work mailboxes. Maybe another with different formatting for sending email to cell phones and pager devices. Lets just call this one our “Default SMTP Channel”
Click Add, and type in the FQDN of your SMTP server(s). This can be an actual SMTP enabled mail server, or a load balanced virtual name.
I am going to select “Windows Integrated” for my Authentication mechanism, since my mail server does not allow Anonymous connections.
For the Return Address – I have created an actual mail enabled user to send Email notifications through SCOM. This might not be a requirement to be a real mail address – mostly that depends on your mail server security policies.
Next up is the email format. We can customize this with very specific information that is relevant to how we want emails to look from SCOM. I will just accept the defaults for now. I can always come back and customize this one, or create additional channels with different formats later.
The Subscriber:
Next up – creating the subscriber. Right Click “Subscribers” and choose “New Subscriber”
This will default to show your domain account. You can change this to whatever you like:
Next – we need to choose when Kevin wants to receive email notifications. This is especially important for things like on call pager devices, or when people work shifts and only want to see emails during certain times.
Next – we need to add an email address to the subscriber. I will add my default work email:
Then select the Channel type, and the email address:
Additionally – you can configure a specific schedule for this specific address. The previous schedule was for the subscriber itself, but a subscriber can have multiple addresses with different schedules if needed. I will keep things simple and choose “Always send”. Click Finish a couple times and your subscriber is set up.
The Subscription:
Now we create a new subscription – Right Click “Subscriptions” and choose New Subscription.
Give your subscription a descriptive name that describes what it is and who it is to. Like – “Messaging team – all critical email alerts” Here is mine:
On the criteria screen – we have some very granular capabilities to scope this subscription. My goal for this simple one is just to send me any new critical alert that comes into my environment:
Next we add the subscribers to the subscription:
We also need to choose which Channel we want to use for this subscription:
On this same screen – there is an option for delay aging:
What that does – is allow for you to have multiple alert subscriptions – and using delay – create an escalation path if an alert is not modified in a way that takes it out of the notification path for these subscriptions.
Click “Finish” and we are all set. Behind the scenes – what happened is that all this information was actually written to a special management pack – the Microsoft.SystemCenter.Notifications.Internal MP.
Let’s test our work.
I have a test rule that generates a critical alert whenever a specific event is written to the event log. Since I subscribed to all critical alerts – this should trigger my subscription and deliver an email:
It worked!
Advanced configuration – setting up a Run As Account to authenticate to the SMTP server:
Note – there is a Run-As Profile that ships with SCOM called the “Notification Account”. If this is not configured, SCOM will try to authenticate to the Exchange server using the Management Server Action Account. If this is not allowed to authenticate, you might need to configure this Run-As profile with a Run As Account.
For instance – I disabled the ability for mail relay on my Exchange server. When I do this – only mail enabled Exchange servers can connect to it. Subsequent notifications fail to go through – and I will see two possible alerts in the console:
Failed to send notification
Notification subsystem failed to send notification over ‘Smtp’ protocol to ‘kevinhol@opsmgr.net’. Rule id: Subscription02e8b6be_528d_407c_8edf_5f29dddaae6b
Failed to send notification using server/device
Notification subsystem failed to send notification using device/server ‘ex10mb1.opsmgr.net’ over ‘Smtp’ protocol to ‘kevinhol@opsmgr.net’. Microsoft.EnterpriseManagement.HealthService.Modules.Notification.SmtpNotificationException: Mailbox unavailable. The server response was: 5.7.1 Client does not have permissions to send as this sender. Smtp status code ‘MailboxUnavailable’. Rule id: Subscription02e8b6be_528d_407c_8edf_5f29dddaae6b
In this case – I must configure the Run-As account with a credential that is able to authenticate properly with my Mail Server. I already have a user account and mailbox set up: OPSMGR\scomnotify
Under Administration > Run As Configuration > Accounts – create a Run As Account.
The account type will be “Windows” and give it a name that makes sense:
Input the user account credentials:
Choose “More Secure” and click Next, then Close.
So – we have created our Run As Account – next we need to choose where to distribute it. Account credential distribution is part of the “More Secure” option – we need to choose which Health Services will be allowed to use this credential. In this case – we want to distribute the account to the management server pool in SCOM 2012 that handles notifications.
Open the properties of our newly created action account, and select the Distribution tab:
Click “Add”, and in the Option field – change it to “Search by Resource Pool Name” and click Search:
Choose the Notifications Resource Pool, click Add, and OK:
Now we have created our Run As Account for notifications, and then distributed it to the Notifications Resource Pool (which contains all management servers dynamically)
Next – we need to configure the Run As Profile – which will associate this account credential with the actual Notification workflows.
Under Administration > Run As Configuration > Profiles, find the “Notification Account” profile. Open the properties of this Profile.
Under Run As Accounts – click Add:
Select our Notification Run As Account, and click OK
Then Save it. This will update the Microsoft.SystemCenter.SecureReferenceOverride MP with these credentials and configurations for notification workflows.
From this point forward – Whichever Management server in the Notifications Resource Pool that is currently responsible for handling notifications, will spawn a MonitoringHost.exe process under our credential that we configured:
This credential will be used to authenticate to the Exchange server to send SMTP notifications. Now my email notifications are flowing smoothly once again! If the current management server goes down, another management server in the Notifications Resource Pool will pick up this responsibility and spawn the process, and continue sending notifications.
High availability out of the box. One of the benefits of the improved SCOM 2012 architecture improvements.
Hi
Many thanks for your article.
Is there anyway to define a global email rule for all monitored equipment regardless type of the equipment? When there is no answer to ping an email shall be send to the servicedesk.
Is it possible to make more complex rules such as the one that follows:
– A monitored equipment stopped responding to ping.
– SCOM keeps trying to ping for a period of time.If response ok then clear alarm else send an email to servicedesk.
– When servicedesk cleared the problem, SCOM keep an eye on the equipment for some period of time to check it is functioning properly.
Many thanks.
Kind regards,
Sam
Hello,
Working in SCOM 2019.
For some reason, under the Subscription Scope, I’m on able to see two available conditions..?
::: raised by any instance in a specific group
::: raised by any instance of a specific class
Not exactly sure what would cause this… ?
Any help or explanation would be much appreciated,
Thanks!!
Nevermind…. I didn’t realize that SCOM 2019 separated the “Scope” and “Criteria” into two sections.
Hi,
Could we change the Date Format in SCOM E-Mail notification?
FYI: I have configured SCOM e-mail notifications. SCOM server has Date Format as dd/mm/yyyy, SCOM console shows date in said format too i.e. dd/mm/yyyy, but e-mails sent by notification workflow has date in mm/dd/yyyy format.
is it by design? can we change it somehow?
intresting thought I’ve tried to look at this also for exampel add 24hours format
Hi Kevin,
I guess if you assign a action account to the notifications actions pool. This account is used for all channels, including command channel?
Standard it seems the computer account is used for command channel…
br
Charlez
Yes, the RunAs account for notifications will be used by the notification command channel as well.
In SCOM Notification library from SCOM 2016: Microsoft.SystemCenter.Notifications.Library version 7.2.11719.0
Command channel:
SMTP example:
This was the same in:
SCOM 2012 R2, version 7.1.10226.0
SCOM 2012 RTM version 7.0.8560.0
You have to go all the way back to SCOM 2007R2 – where this did not have a RunAs profile association for the command channel.
Without a RunAs profile association – this workflow will run as the default action account, which on management servers is the Management Server action account. Our documentation was always wrong, when it referenced Local System. That would only ever be a true statement, IF the Management Server Action account was using Local System, AND there was no RunAs profile associated to a notification RunAs account.
Hi Kevin,
You said : “Yes, the RunAs account for notifications will be used by the notification command channel as well”
I need that to work and I am trying to reproduce it, but unfortunatly whatever settings I put as run as account, the command channel (powershell.exe execution) does start with the local system account.
Do you have any clue how to make that work ?
Cheers,
Thibaut
What is your management server action account where the command channel is running?
Did you associate the runas account to the Notifications RunAs profile using all targeted objects?
Did you distribute that runas account to the notifications resource pool?
Does the notification account have rights on the management server to log on locally, as a service, or is it throwing an error in the event log on the management server when it tries to log on?
Hi Kevin,
A customer of mine made a request where they would like to receive hourly email notification updates for a specific service that is down.
So if a specific service went down at 5pm, they want to receive an email notification that it went down. Then if the service is still down an hour later, they want to receive another email notification. They want to receive hourly notifications until the service is back in the running state.
Is this possible?
You would need to use a custom script – to get specific alerts, and then update them in some way that triggers a new notification….. that runs hourly.
Nagging is a terrible practice and not a good way to have a mature ITSM process…. customers that want this are typically not mature and willfully trying to stay that way. 🙁
I thought the same.
They already receive a notification when the alert is raised, I don’t understand why they would want another one every hour. We have it set up to auto resolve the service if it ever goes down, and then once the issue is resolved they receive another email notification that the service is running again. I will speak to them and let them know that it doesn’t make sense to do it the way they requested.
Thanks, Kevin!
Hello Kevin,
Is there a way to filter alerts for a specific scope? i.e. right now I have a scope for the SCOM Mgt server and the database/DW. Can I filter the alerts to send notifications only for specific issues, for example when there is no heartbeat, instead of sending notification for every alert that has those servers as a path?
Hi Kevin,
Thanks for the Article, I followed the steps, however the MonitoringHost.exe with the configured user name has not appeared yet? Is this a case for a simple restart of severs or SCOM services? Is there anything I can do to troubleshoot?
Hi Kevin, in our environment we are not using any Notification Action Account. We have configured command channel notifications to trigger Powershell script when it detects the Scom rule. Its working amazingly. But out of 6 Management Servers (MSs) (where all of them are in Notification Resource Pools), ONLY 1 Management Server is always performing the checks defined in the Powershell Script. We would like to know, why only specific Management Server is being selected for checks. If the MS which is taking precedence to run the PS script, if it goes down, does any one of the remaining MS Servers take over in running the PowerShell script via command channel?
That’s by design. There is a single notification workflow, which is assigned to the notification server. It only runs on ONE server, and will fail over to any available server in the pool. This is why I like to just put two servers in my notification pools, so if I need to troubleshoot I know it is running on one or the other.
When configuring Notification Subscribe on SCOM 2019, I get Failed to save the Notification Subscribe: The operation for this recipient could not be completed.
Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.
System.ArgumentException: The operation for the recipient could not be completed. See inner exception for details. —> Microsoft.EnterpriseManagement.Common.ManagementPackException: Database error. MPInfra_p_ManagementPackInstall failed with exception:
Database error. MPInfra_p_ManagementPackInstall failed with exception:
[SQL Error Code: -2146232060][MP ID: dcfdedc4-68bd-42b4-1e9a-ba94b1577732][MP Version: 10.19.10050.0][MP PKT: ] Procedure or function p_MPImportXML has too many arguments specified.
at Microsoft.EnterpriseManagement.Common.Internal.ServiceProxy.HandleFault(String methodName, Message message)
at Microsoft.EnterpriseManagement.Common.Internal.MonitoringNotificationServiceProxy.UpsertNotificationRecipient(Guid recipientId, String name, String schedulesAndDevicesXml, Boolean isUpdate)
at Microsoft.EnterpriseManagement.MonitoringNotificationManagement.InsertNotificationRecipient(NotificationRecipient notificationRecipient)
— End of inner exception stack trace —
at Microsoft.EnterpriseManagement.MonitoringNotificationManagement.InsertNotificationRecipient(NotificationRecipient notificationRecipient)
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Notification.RecipientCompletionPage.CommitChanges(Object sender, DoWorkEventArgs e)
: Database error. MPInfra_p_ManagementPackInstall failed with exception:
Database error. MPInfra_p_ManagementPackInstall failed with exception:
[SQL Error Code: -2146232060][MP ID: dcfdedc4-68bd-42b4-1e9a-ba94b1577732][MP Version: 10.19.10050.0][MP PKT: ] Procedure or function p_MPImportXML has too many arguments specified.
What could be the problem?
This happened when someone patches the server with an update roll up but failed to ensure the database was updated. You should ensure UR3 is correctly applied then manually apply the SQL scripts to the databases.
Dear Kevin
In fact I am not writing to you not on this subject
I have a question about SQL
I run a query in POWERSHELL and the query has two commas
One wraps the query and the other inside to configure DB
But I get an error
What do I need to do to make the query run without errors?
Just put the query in quotes. Multiple commas are a non-issue. It is more likely your SQL query has disallowed XML special characters like <>
Dear Kevin,
thanks a lot for all your posts and sorry for my bad english.
My Question is about authenticated Mail:
We have the following environment:
Domain A: SCOM MMS, SQL
Domain B: Gateway, Agents and the Exchange-Server we should use
The domains are not trusted, so I can’t use the User for authenticated Mails for the standard notification ressource pool.
I made the Notification Ressource Pool manually managed, removed the ManagementServer and added the 2 Gateways we have in Domain B.
I prepared a Notification Action Account like you discribed above and delivered this account to the Notification Ressource Pool.
On the Gateways I see, the Ressource Pool is used and activated an both GW-Servers.
I expected but didn’t don’t see a service or process on the GW-Servers using the Notification Action Account,
and also no mails be send (simple subscription for all new alerts to just my account).
The exchange guys didn’t see any try of the GW-Servers to send mail.
I think, my construct will not be working properly at all, because there will also no other subscription will work, even if the authenticated mail will work (what is not the case right now).
Maybe I am on the very wrong way.
So my question in short is, is there a possibility to send only subscription mails via a gateway in a untrusted domain (with a user from that untrusted domain) and use the Management Servers for all other Subscription channels like command for example?
The Notification Action Account I wanted to use have Log On Locally and Log On As A Service rights on the Gateway Servers, btw.
Thanks in advance for any suggestion!
As additional Information: we use the Notification Action Account-User with powershell in a timed command successfully from the RMSE, btw, so there should be a way…but I am lost… 🙁
There is a new step with this problem:
I can see the process created on the gateway with the User from domain B.
But very soon after that process is created I get an error with eventid 4508 from HealthService:
Loading managed module type in assembly “Microsoft.Mom.ValidateAlertSubscriptionDataSourceModule, Culture=””, PublicKeyToken=”31bf3856ad364e35″, Version=”7.0.5000.0″” with type name “Microsoft.EnterpriseManagement.Mom.ValidateAlertSubscriptionModule.ValidateAlertSubscriptionDataSource” failed with error “The module assembly “Microsoft.Mom.ValidateAlertSubscriptionDataSourceModule, Culture=””, PublicKeyToken=”31bf3856ad364e35″, Version=”7.0.5000.0″” could not be loaded. The exception was: \nCould not load file or assembly ‘Microsoft.Mom.ValidateAlertSubscriptionDataSourceModule, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified..”.
Notification pools need to use management servers. There are many modules required only present on a MS.
Ah, ok, thats what I was afraid of…at least its clear now.
Thanks!
Hello Kevin,
Is there a way to configure an external account as run as account?
let’s say scom server is in microsoft.net domain and SMTP is on micro.net. And i have created the run as account also on micro.net and has return address as john.doe@micro.com.
I can communicate to the smtp server in micro.net on port 25. But when i configured the run as account on scom as p.smtp-scom@micro.net.. getting below error:
“The health service could not log on the runas account micro.net\p.smtp-scom from management group “Micro-SCOM” [This is my management group]. The error is the username or password is incorrect(1326L). This will prevent the health service from monitoring or performing actions using this RunAs account.
i have alerts from all the servers which are in management servers pool.. “Unable to verify Run As account”
This could be because i am using an external account. Do you have any suggestions?
i managed to made it work.. in my case i don’t need any “run as account” to be configured.. Just Notification account and set it to use “Local system Windows account” as Run as account.