What account will command channel notifications Run As in SCOM?

I get this question several times a year. Let’s find out for sure.

Create a new Command Channel.

For the configuration settings, I will use the following, as this will allow me to write all common alert fields to a logfile:

Full Path of the command file:

%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe

Command line parameters

-executionPolicy bypass -noprofile -file "C:\scripts\logalert.ps1" -AlertID "$Data[Default='NotPresent']/Context/DataItem/AlertId$" -AlertName "$Data[Default='NotPresent']/Context/DataItem/AlertName$" -AlertDescription "$Data[Default='NotPresent']/Context/DataItem/AlertDescription$" -Severity "$Data[Default='NotPresent']/Context/DataItem/Severity$" -ResolutionState "$Data[Default='NotPresent']/Context/DataItem/ResolutionState$" -ManagedEntityPath "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityPath$" -ManagedEntityDisplayName "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityDisplayName$" -ManagedEntityFullName "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityFullName$" -ManagedEntity "$Data[Default='NotPresent']/Context/DataItem/ManagedEntity$" -CreatedByMonitor "$Data[Default='NotPresent']/Context/DataItem/CreatedByMonitor$" -WorkflowId "$Data[Default='NotPresent']/Context/DataItem/WorkflowId$" -DataItemCreateTimeLocal "$Data[Default='NotPresent']/Context/DataItem/DataItemCreateTimeLocal$" -WebConsoleUrl "$Target/Property[Type="Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer"]/WebConsoleUrl$?DisplayMode=Pivot&AlertID=$UrlEncodeData/Context/DataItem/AlertId$"

Startup folder for the command line:

C:\scripts

In C:\scripts I place my script file “logalert.ps1”:

Logalert.ps1 contents:

param($AlertID, $AlertName, $AlertDescription, $Severity, $ResolutionState, $ManagedEntityPath, $ManagedEntityDisplayName, $ManagedEntityFullName, $ManagedEntity, $CreatedByMonitor, $WorkflowId, $DataItemCreateTimeLocal, $WebConsoleUrl)

$ScriptName = "logalert.ps1"
$EventID = "1234"

$whoami = whoami

$momapi = New-Object -comObject MOM.ScriptAPI
#Log script event that we are starting task
$momapi.LogScriptEvent($ScriptName,$EventID,0,"`nScript is starting. `nRunning as ($whoami). `nAlertID: ($AlertID) `nAlertName: ($AlertName) `nAlertDescription:($AlertDescription) `nSeverity:($Severity) `nResolutionState: ($ResolutionState) `nManagedEntityPath: ($ManagedEntityPath) `nManagedEntityDisplayName: ($ManagedEntityDisplayName) `nManagedEntityFullName: ($ManagedEntityFullName) `nManagedEntity: ($ManagedEntity) `nCreatedByMonitor: ($CreatedByMonitor) `nWorkFlowId: ($WorkflowId) `nDataItemCreateTimeLocal: ($DataItemCreateTimeLocal) `nWebConsoleURL: ($WebConsoleUrl)")

Now lets set up a subscriber for command channel alerts, and a subscription:

Create a new subscriber and call it “Command Channel Subscriber”

Add a new subscriber address, like so:

Next, add the subscriber to a subscription. I will subscribe to my Test Alerts from my SCOM.Management pack:

Add the Command Channel Subscriber to the subscription:

Add the proper Channels:

And finish:

Next – lets review our resource pools.

For testing, I like to change the Notifications pool to manual, and just put a single server in there temporarily. This will ensure I know which server is running ALL notification.

Then I will trigger the Create Test Event task from the SCOM.Management pack:

Which will trigger a test Alert:

Now open the Operations Manager event log on the Management server in our resource pool, and we should see a 1234 Event, which was triggered by the command channel script.

I can see – that by default – the account used for Notifications is the same as the Management Server Action Account for this management server.

Now – lets configure the Notification RunAs account:

Create a new RunAs account for Notifications (this needs to be a real account in Active Directory):

Distribute this account to the Notifications Resource Pool:

Next, we must ASSOCIATE the RunAs account to the Notifications Account Profile. Browse to the Profiles, and find the Notifications Account Profile:

Associate your Notifications RunAs account PROFILE with your newly created RunAs account, for all targeted objects:

Once this association is made, you will see a new MonitoringHost.exe process spun up on the Management server, under your new account:

All notifications will now execute as this account. You might need to grant this account privileges on your management server for it to be able to Log On Locally, or Log On As A Service, depending on your SCOM version. You can review the management server event log for errors which will show up if there is a configuration issue.

Now – go back and trigger a new alert

And check the log:

We can easily see the Notification Command Channel runs as our new Notification RunAs account.

Now lets review the XML to understand how or why this works:

When you create a Notification Subscription, you are actually creating a RULE in the unsealed Microsoft.SystemCenter.Notifications.Internal management pack.

Here is an example:

      <Rule ID="Subscription8568d5a5_a6d3_4f45_a8fd_14d42eebe276" Enabled="true" Target="Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100">
        <Category>Notification</Category>
        <DataSources>
          <DataSource ID="DS1" RunAs="SystemCenter!Microsoft.SystemCenter.DatabaseWriteActionAccount" TypeID="SystemCenter!Microsoft.SystemCenter.SubscribedAlertProvider">
            <AlertChangedSubscription Property="Any">
              <Criteria>
                <Expression>
                  <SimpleExpression xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
                    <ValueExpression>
                      <Property>RuleId</Property>
                    </ValueExpression>
                    <Operator>Equal</Operator>
                    <ValueExpression>
                      <Value>8060f053-dc84-330d-c7d0-3c659f9b7ca7</Value>
                    </ValueExpression>
                  </SimpleExpression>
                </Expression>
              </Criteria>
              <ExpirationStartTime>03/05/2021 16:25:37</ExpirationStartTime>
              <PollingIntervalMinutes>1</PollingIntervalMinutes>
              <UserSid>S-1-5-21-3626055071-1639654894-2113106914-1104</UserSid>
              <LanguageCode>ENU</LanguageCode>
              <ExcludeNonNullConnectorIds>false</ExcludeNonNullConnectorIds>
              <RuleId>$MPElement$</RuleId>
              <TargetBaseManagedEntityId>$Target/Id$</TargetBaseManagedEntityId>
              <TimeZone>6801000000000000C4FFFFFF00000B0000000100020000000000000000000300000002000200000000000000|Central Standard Time</TimeZone>
            </AlertChangedSubscription>
          </DataSource>
        </DataSources>
        <ConditionDetection ID="CD1" TypeID="Microsoft.SystemCenter.Notification.Recipients">
          <Recipients>
            <To>
              <DirectoryReference>
                <RecipientId>646e126c-c96e-479f-b555-b9df1386ab59</RecipientId>
              </DirectoryReference>
              <DirectoryReference>
                <RecipientId>187dfeec-6197-4ca5-9272-d6b0aa2c23f3</RecipientId>
              </DirectoryReference>
            </To>
          </Recipients>
        </ConditionDetection>
        <WriteActions>
          <WriteAction ID="WA0" TypeID="Cmd_b8d0e358_b9b8_468b_b7de_38ba028a6c7b" />
          <WriteAction ID="WA1" TypeID="SMTPAction_6cb98307_eb31_4f10_b35a_31f1e7648807" />
        </WriteActions>
      </Rule>

If you scroll down to the Write Actions, you can see our “Cmd_” write action which is the command channel. Look further up in the XML of the MP and you can find this write action definition. This Write Action is our CHANNEL.

      <WriteActionModuleType ID="Cmd_b8d0e358_b9b8_468b_b7de_38ba028a6c7b" Accessibility="Public" Batching="false">
        <Configuration />
        <ModuleImplementation Isolation="Any">
          <Composite>
            <MemberModules>
              <WriteAction ID="NotificationAction" TypeID="Notification!Microsoft.SystemCenter.Notification.CommandAction">
                <RecipientProtocol>Cmd_b8d0e358_b9b8_468b_b7de_38ba028a6c7b</RecipientProtocol>
                <ApplicationName>%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe</ApplicationName>
                <WorkingDirectory>c:\scripts</WorkingDirectory>
                <CommandLine>-executionPolicy bypass -noprofile -file "C:\scripts\logalert.ps1" -AlertID "$Data[Default='NotPresent']/Context/DataItem/AlertId$" -AlertName "$Data[Default='NotPresent']/Context/DataItem/AlertName$" -AlertDescription "$Data[Default='NotPresent']/Context/DataItem/AlertDescription$" -Severity "$Data[Default='NotPresent']/Context/DataItem/Severity$" -ResolutionState "$Data[Default='NotPresent']/Context/DataItem/ResolutionState$" -ManagedEntityPath "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityPath$" -ManagedEntityDisplayName "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityDisplayName$" -ManagedEntityFullName "$Data[Default='NotPresent']/Context/DataItem/ManagedEntityFullName$" -ManagedEntity "$Data[Default='NotPresent']/Context/DataItem/ManagedEntity$" -CreatedByMonitor "$Data[Default='NotPresent']/Context/DataItem/CreatedByMonitor$" -WorkflowId "$Data[Default='NotPresent']/Context/DataItem/WorkflowId$" -DataItemCreateTimeLocal "$Data[Default='NotPresent']/Context/DataItem/DataItemCreateTimeLocal$" -WebConsoleUrl "$Target/Property[Type="Notification!Microsoft.SystemCenter.AlertNotificationSubscriptionServer"]/WebConsoleUrl$?DisplayMode=Pivot&amp;AlertID=$UrlEncodeData/Context/DataItem/AlertId$"</CommandLine>
                <TimeoutSeconds>2147483647</TimeoutSeconds>
                <RequireOutput>false</RequireOutput>
              </WriteAction>
            </MemberModules>
            <Composition>
              <Node ID="NotificationAction" />
            </Composition>
          </Composite>
        </ModuleImplementation>
        <InputType>Notification!Microsoft.SystemCenter.Notification.RecipientsData</InputType>
      </WriteActionModuleType>

This CHANNEL uses the Microsoft.SystemCenter.Notification.CommandAction write action from the Microsoft.SystemCenter.Notifications.Library, as seen in the XML above.

If we review that Microsoft.SystemCenter.Notification.CommandAction write action XML, we can see the RUNAS association where this workflow WILL use the Notifications RunAs account profile:

      <WriteActionModuleType ID="Microsoft.SystemCenter.Notification.CommandAction" Accessibility="Public" RunAs="Microsoft.SystemCenter.Notification.NotificationActionAccount" Batching="false">
        <Configuration>
          <IncludeSchemaTypes>
            <SchemaType>System!System.CommandExecuterSchema</SchemaType>
          </IncludeSchemaTypes>
          <xsd:element name="RecipientProtocol" type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
          <xsd:element name="ApplicationName" type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
          <xsd:element name="WorkingDirectory" type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
          <xsd:element name="CommandLine" type="xsd:string" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
          <xsd:element name="TimeoutSeconds" type="xsd:integer" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
          <xsd:element name="RequireOutput" type="xsd:boolean" xmlns:xsd="http://www.w3.org/2001/XMLSchema" />
        </Configuration>
        <ModuleImplementation Isolation="Any">
          <Composite>
            <MemberModules>
              <ConditionDetection ID="RecipientFilter" TypeID="Microsoft.SystemCenter.Notification.RecipientFilter">
                <Protocol>$Config/RecipientProtocol$</Protocol>
                <Multicast>false</Multicast>
              </ConditionDetection>
              <WriteAction ID="Endpoint" TypeID="System!System.CommandExecuter">
                <ApplicationName>$Config/ApplicationName$</ApplicationName>
                <WorkingDirectory>$Config/WorkingDirectory$</WorkingDirectory>
                <CommandLine>$Config/CommandLine$</CommandLine>
                <TimeoutSeconds>$Config/TimeoutSeconds$</TimeoutSeconds>
                <RequireOutput>$Config/RequireOutput$</RequireOutput>
              </WriteAction>
            </MemberModules>
            <Composition>
              <Node ID="Endpoint">
                <Node ID="RecipientFilter" />
              </Node>
            </Composition>
          </Composite>
        </ModuleImplementation>
        <InputType>Microsoft.SystemCenter.Notification.RecipientsData</InputType>
      </WriteActionModuleType>

Specifically this part, on the first line:

That is what tells this workflow to load as the account associated to that RunAs profile, if one is associated.

Lastly, remember, if you are going to use Command Channel Notifications (which I generally do not recommend)…. these have a scaling issue. By default, SCOM will only allow 5 simultaneous responses (5 alerts with the command channel subscribing to them) in a single run, and will drop the rest. If this happens, you might see the following event:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          8/25/2021 7:48:16 AM
Event ID:      21410
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SCOM1.opsmgr.net
Description:
The process could not be created because the maximum number of asynchronous responses (5) has already been reached, and it will be dropped.

Command executed:    “C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe” -executionPolicy bypass -noprofile -file “C:\scripts\logalert.ps1” -AlertID “{addd5be7-b31c-4a0c-911d-cfc3d7bd2a5f}” -AlertName “Test Alert on Test Event 100” -AlertDescription “This is a test alert fired by event ID 100: \n Event Description: \n This is a Test event 100” -Severity “0” -ResolutionState “0” -ManagedEntityPath “SCORCH16.opsmgr.net” -ManagedEntityDisplayName “SCORCH16.opsmgr.net” -ManagedEntityFullName “SCOM.Management.Class:SCORCH16.opsmgr.net” -ManagedEntity “{efeeade0-eb50-e531-96f8-2887251c429e}” -CreatedByMonitor “false” -WorkflowId “{8060f053-dc84-330d-c7d0-3c659f9b7ca7}” -DataItemCreateTimeLocal “8/25/2021 7:48:16 AM” -WebConsoleUrl “http://SCOM1/OperationsManager?DisplayMode=Pivot&AlertID=$UrlEncodeData/Context/DataItem/AlertId$”
Working Directory:    c:\scripts

One or more workflows were affected by this.

Workflow name: Subscription8568d5a5_a6d3_4f45_a8fd_14d42eebe276
Instance name: Alert Notification Subscription Server
Instance ID: {E07E3FAB-53BC-BC14-1634-5A6E949F9230}
Management group: SCOM

And an alert will be generated:

To work around this you need to set a Registry key/value to allow for the maximum number of asynchronous responses to alerts,which can be set as high as 100:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Modules\Global\Command Executer\AsyncProcessLimit

Create a new REG_DWORD value. Valid options are from 5 to a maximum of 100.

This will allow 100 scripts (or command channel responses) to execute simultaneously. Take care that 100 of these will not harm the resources of your management server, and understand that if more than 100 alerts arrive (in a single minute) that need this response run, any over the first 100 will be dropped. They will not queue and will never have the command channel run. It is important to understand this limitation of using the command channel, especially for something like a ticketing system integration.

5 Comments

michael
August 30, 2021 at 4:12 am

our managementgroups is quite large with many subscriptions, Do you now of any way of extending async limit to beyond 100, the load wouldn’t be an issue.
- Kevin Holman Post author
  August 30, 2021 at 6:33 am
  
  I don’t know any way to do that. We generally recommend moving to Orchestrator or use the product connector framework at that point, not command channels.
Dimitri
August 24, 2022 at 2:42 am

I have absolutely no way to make it work. I checked all forums from MS and others. The script simply is not starting.
No matter what I put in command line, nothing runs.
On some MS forum they say the startup folder for the command line must be the path for powershell.exe
In some other they say it’s the path of the script to be run.
I copied the configuration you suggested, but still nothing happens. Is there a way to debug or at least to trace what happens ? I cannot see anything also in event log.

Thanks
Kailasnath Aanchery
January 19, 2023 at 11:25 pm

Hi Kevin

We have new installation of SCOM 2022. I have created a command subscription but the command (a powershell script) is not getting triggered when the alert is logged(thal service heartbeat failure). To test, I created an email subscription for the same alert and that works. The powershell execution policy is proper too. I am able to manually run the script but the scom subsription would not trigger it.
Dimitrov
February 1, 2023 at 4:52 am

Hi Kevin,

The service is not starting with the run as account user.

i always get the error: Microsoft.EnterpriseManagement.HealthService.Modules.Notification.SmtpNotificationException: The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.7.57 SMTP; Client was not authenticated to send anonymous mail during MAIL FROM. Smtp status code ‘MustIssueStartTlsFirst’.

How can i fix this?

What account will command channel notifications Run As in SCOM?

5 Comments

Leave a Reply Cancel reply