Automating Run As Account Distribution – Finally!

I wrote a post explaining Run As accounts a while back here: https://kevinholman.com/2010/09/08/configuring-run-as-accounts-and-profiles-in-opsmgr-a-sql-management-pack-example/

One of the biggest pain points in distributing RunAs accounts, is managing the Health Service distribution of the credential. When you associate a RunAs profile to an account, this tells agents that are covered by the association, to try and use the RunAs account for any workflows where it is called for. However, this will fail if you haven’t distributed the account first, and will result in Alerts being generated like the following:

System Center Management Health Service Credentials Not Found Alert Message

An account specified in the Run As profile “Microsoft.SQLServer.SQLProbeAccount” cannot be resolved.

These are important alerts because they tell you when a HealthService has been associated in a profile, but is missing the credential distribution.

Previously – you had to manually find these accounts, and manually distribute the credential to the HealthService. This was a maintenance intensive process.

Over the years, there have been posted a few examples to blogs about how to automate this. Most of them dealt with getting these alerts, then responding with a script to distribute the RunAs account to the MonitoringObject that the alert was generated from.

I never liked this approach, because it was error prone to me. It is possible you made a mistake in your profile association, and over-scoped the computers which should try and load the RunAs account. Using alerts also mean that you’d never see which servers got a distribution, so you would not be aware of a misconfiguration. Additionally, you might see a large number of “Log on Locally” failed alerts from any agent where the credential was distributed, but the credential doesn’t have rights.

Lastly – my customers often need to customize where the account would be distributed. Perhaps, in the example of SQL, the IT DBA team only supported a portion of the total SQL servers discovered by SCOM. Or perhaps they needed multiple different RunAs accounts for different SQL servers. They would likely create groups of Windows Computer objects to represent these different SQL servers….. so what I often propose is to use the GROUP to determine RunAs account distribution.

This group methodology allows for a completely hands-free configuration of your RunAs account distribution, and you can delegate the group membership to the respective teams, perhaps by using a Registry key for them to determine group membership.

So that brings us to using groups instead of alerts. There are actually a few blog posts out there on how to do this, however I always found them to be good base example scripts, but lacking in features needed. For instance, what if our group contained a Cluster virtual server, such as the “SQL Computers” group often will…. we need to distribute the RunAs credential to the NODE health services, not the virtual. The script needs to account for that.

Additionally, many of the scripts will simply REPLACE the RunAs distribution when the run. However, there will almost always be one-off scenarios where you need to quickly add a HealthService to the distribution, even though it might now reside in the core group. Therefore the script should provide for a way to gather the existing list of distributed health services, and only add news ones where necessary.

With the collaboration of several Microsoft support PFE’s (Matt Taylor, Scott Murray, Mark Manty, Tim McFadden, and Russ Slaten) here is a script that performs this task.

The PowerShell script has a function, and the only thing you will need to customize is at the end of the script, just modify the line calling the function DistributeRunAsAccounts with your RunAs account name and Group name. If you have multiple accounts to distribute, just add another line and call the function again as you see in my example.

The Account display name and Group display name are CASE SENSITIVE!

The group used should contain WINDOWS COMPUTER objects.

#=================================================================================
#  Health Service Run As Account by Group Distribution Script
#  by Kevin Holman
#
#  This script takes a SCOM group of WINDOWS COMPUTERS by name, and distributes a RunAs account
#  to all Healthservice members of the group including nodes of clusters
#
#  Version 2.1
#=================================================================================


# Manual Testing section - put stuff here for manually testing script - typically parameters:
#=================================================================================
#=================================================================================


# Constants section - modify stuff here:
#=================================================================================
# Assign script name variable for use in event logging
$ScriptName = "Example.RunAsDistributor.Rule.ps1"
$EventID = "3250"
#=================================================================================


# Starting Script section - All scripts get this
#=================================================================================
# Gather the start time of the script
$StartTime = Get-Date
#Set variable to be used in logging events
$whoami = whoami
# Load MOMScript API
$momapi = New-Object -comObject MOM.ScriptAPI
#Log script event that we are starting task
$momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Script is starting. `n Running as ($whoami).")
#=================================================================================


# Connect to local SCOM Management Group Section - If required
#=================================================================================
# I have found this to be the most reliable method to load SCOM modules for scripts running on Management Servers
# Clear any previous errors
$Error.Clear()
# Import the OperationsManager module and connect to the management group
$SCOMPowerShellKey = "HKLM:\SOFTWARE\Microsoft\System Center Operations Manager\12\Setup\Powershell\V2"
$SCOMModulePath = Join-Path (Get-ItemProperty $SCOMPowerShellKey).InstallDirectory "OperationsManager"
Import-module $SCOMModulePath
New-DefaultManagementGroupConnection "localhost"
IF ($Error) 
{ 
  $momapi.LogScriptEvent($ScriptName,$EventID,1,"`n FATAL ERROR: Unable to load OperationsManager module or unable to connect to Management Server. `n Terminating script. `n Error is: ($Error).")
  EXIT
}
#=================================================================================


# Begin MAIN script section
#=================================================================================
# Begin Function
Function DistributeRunAsAccounts ($ActRunAsDisplayName, $DistGroupName)
{
  #Get the SCOM Run As account by display name
  $ActRunAs = Get-SCOMRunAsAccount $ActRunAsDisplayName
  $ActRunAsDomain = $ActRunAs.Domain
  $ActRunAsUser = $ActRunAs.UserName
  $momapi.LogScriptEvent($ScriptName,$EventID,0,"`n The main function is starting. `n RunAs Account Display Name: ($ActRunAsDisplayName). `n Group Display Name: ($DistGroupName). `n RunAs Account Domain: ($ActRunAsDomain). `n RunAs Account Username: ($ActRunAsUser).")
  #Choose a group by displayname.  
  #This group contains Windows Computers objects for agent managed computers and for cluster virtual server names.
  [array]$DistComputerNames = Get-SCOMGroup -DisplayName $DistGroupName | Get-SCOMClassInstance | Select-Object -Property DisplayName
  $Distcompnamecount = $DistComputerNames.count
  $momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Items in Group is: ($Distcompnamecount) for account: ($ActRunAsDisplayName) and group: ($DistGroupName).")
  #Exit script and log error if group returns no valid instances of Windows Computer property displayname
  IF ($DistComputerNames.count -lt 1)
  {
    #Log an event that our script is ending in error
    $momapi.LogScriptEvent($ScriptName,$EventID,1,"`n FATAL ERROR: `n The group was not found or contained no objects. `n Account: ($ActRunAsDisplayName) and Group: ($DistGroupName). `n Terminating script.")
    EXIT
  }
  #Set DistAgents to empty
  $DistAgents = @()
  #Get an array of Health Service Objects which match the displayname of the group membership objects
  Foreach ($DistComputerName in $DistComputerNames)
  {
    $DistAgents += $HealthServiceClass | where {$_.DisplayName -eq $DistComputerName.DisplayName} 
  }
  IF ($DistAgents.count -gt 0)
  {
    #Compare lists (diff) to get a list of all Computers in the group that do not have a matching HealthService.  
    #Assume these are Network Name values for clustered resource groups
    $DistClusters = Compare-Object $DistComputerNames.DisplayName $DistAgents.DisplayName -PassThru
  }
  ELSE
  {
    #Assume that the group only contained Windows Cluster objects and no agents.
    $DistClusters = $DistComputerNames
  }
  # If there are no clusters in the group skip the cluster node section of the script
  IF ($DistClusters.count -ge 1)
  {
    #Set UknownObject array equal to null to ensure it is empty from any previous script runs
    $UnknownObjects = @()
    #Get the relationship ID we need to find nodes that host a cluster
    $rid = Get-SCOMRelationship -DisplayName 'Health Service manages Entity'
    #Get all Cluster virtual server class instances in an array
    $vsclass = Get-SCOMClass -name 'Microsoft.Windows.Cluster.VirtualServer' | Get-scomclassinstance
    #Create a loop to find the node names of each cluster and add those names to an array
    foreach ($clname in $DistClusters)
    {
       #write-host 'clname: '$clname
       #Get the Virtual Server class instance for each cluster name
       $vs = $vsclass | where-object {$_.DisplayName -eq $clname}
       #Continue with the script if we got a match for a virtual server class
       IF ($vs.count -gt 0)
       {
         #write-host 'virtual server: '$vs.DisplayName
         #Get the nodes in an array which have a health service relationship managing the virtual server cluster name
         $nodes = Get-SCOMRelationshipInstance -TargetInstance $vs | Where-Object {$_.relationshipid -eq $rid.id} | Select-Object -Property SourceObject
         #write-host 'nodes: '$nodes.sourceobject
         #Get an array of SCOM Agent objects which match the displayname of objects in the nodenames array and check first to ensure no duplicates
         Foreach ($node in $nodes)
         {
           If ($DistAgents -notcontains $node.SourceObject)
           {
             $DistAgents += $HealthServiceClass | where {$_.DisplayName -eq $node.SourceObject}
           }
         }
       }
       #If objects in the group do not match the displayname for a virtual server class nor a healthservice displayname add them to unknown objects array
       ELSE
       {
         $UnknownObjects += $clname
       }   
    }
  }
  #Get the current RunAs account distribution as it exists today and save it in an array
  #Comment out this entire section if you want to ignore the current distribution and ONLY go with what it in the group
  $ActRunAsDistOld = (Get-SCOMRunAsDistribution $ActRunAs).securedistribution
  #Check and see if there is at least one health service distribution defined - ignore if none found
  #If at least one distribution is found, check for duplicates then add non dupes to the DistAgents array
  IF ($ActRunAsDistOld.count -ge 1)
  {
    Foreach ($ActRunAsOld in $ActRunAsDistOld)
    {
      IF ($DistAgents.DisplayName -notcontains $ActRunAsOld.DisplayName)
      {
        $DistAgents += $ActRunAsOld
      }
    }
  }
  #Set (replace) the RunAs distribution list for the defined RunAs account to whatever we found in DistAgents
  Set-SCOMRunAsDistribution ($ActRunAs) -MoreSecure -SecureDistribution $DistAgents

  #Log event that script is complete, additional variables for logging purposes
  $DistAgentsCount = $DistAgents.count
  $ActRunAsDistOldCount = $ActRunAsDistOld.count
  If ($UnknownObjects.Count -lt 1)
  {
    $UnknownObjects = 'NONE'
  }
  $momapi.LogScriptEvent($ScriptName,$EventID,0,"`n RunAs HealthService Distribution Completed. `n Account displayname: ($ActRunAsDisplayName). `n Group displayname: ($DistGroupName). `n Configured ($DistAgentscount) Healthservice objects. `n Previously there were ($ActRunAsDistOldcount) objects configured. `n Unknown Objects identified: ($UnknownObjects). `n Health service objects added: ($DistAgents).")
}
#=================================================================================
#End Function

#Get all the health service class instances in the management group. 
$momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Getting all HealthService class instances into an array. `n This may take a long time.")
$HealthServiceClass = Get-SCOMClass -DisplayName "Health Service"| Get-SCOMClassInstance
$HSCount = $HealthServiceClass.count
#Log an event for how long this took and how many healthservice objects were returned
$momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Completed getting all HealthService class instances into an array. `n Returned ($HSCount) Health Service objects.")


# Add multiple "Main" lines for each account and group pair that needs dynamic distribution
# THIS IS CASE SENSITIVE!!!
# Example:
#DistributeRunAsAccounts "Your Custom RunAs Account Display Name" "Your Group Display Name"
DistributeRunAsAccounts "Test RunAs Account" "Test RunAs Group"

#=================================================================================
# End MAIN script section


# End of script section
#=================================================================================
#Log an event for script ending and total execution time.
$EndTime = Get-Date
$ScriptTime = ($EndTime - $StartTime).TotalSeconds
$momapi.LogScriptEvent($ScriptName,$EventID,0,"`n Script Completed. `n Script Runtime: ($ScriptTime) seconds.")
#=================================================================================
# End of script

Notes:

Version 1.3 script updated to add error handling of null value arrays when an object in the group doesn’t match any virtual clusters, and added collection of these object names in the final script as unknown objects
Version 1.4 script updated to add error handling when a group contains NO windows computers, or when a group contains only cluster objects and no corresponding agents
Version 1.5 – additional error handling
Version 1.6 – 10/26/15 – edited $RunAs out of script to be able to use this in a MP. $RunAs is a reserved variable in SCOM.
Version 2.0 – 4/3/2016 – significant rewrite to place the main execution in a function, and call it multiple times for multiple accounts, and added better debug logging.
Version 2.1 – 9/10/2017 – rewrite to add error checking, logging, standardize on my latest script template.

You can download the example MP here:

https://github.com/thekevinholman/AutomateDistributeRunAsAccounts

This PowerShell script can be executed by Orchestrator, Task Scheduler, or even SCOM in a Management Pack.

I will show an example of using this for the SQL Run As account. My current RunAs account has not been distributed:

I am using a “vanilla” profile association:

This results in alerts from all my SQL servers where a SQL Role was discovered, and where they try to load a discovery or monitoring workflow that leverages a Runas Account:

When I execute the script – I see the following events in the OpsMgr event log:

Event ID: 3250
Description: RunAsHSDist.ps1 : RunAs HealthService Distribution Script Starting. Executed as (domain\account)

And in the console I see these accounts distributed:

And the previous alerts auto-closed since they came from a Monitor which auto-resolved them when it went back to healthy.

I welcome your feedback if you have any issues.

5 Comments

John Austin
February 19, 2019 at 9:29 am

Hi Kevin,

We have an issue that I’m not sure this script addresses. Our existing script will collect what’s in the distribution currently, put it in an array, update the array with anything new, and then redistribute the credentials accordingly. I *think* this is what your script is doing. The problem: If a weekly agent job fails on an otherwise healthy SQL server that already has the account distributed, there is an alert. If we run the script, it redistributes the account to all SQL servers, causing the health to reset, in turn causing a second alert/incident on that same agent job.

We need to be able to automatically add systems to the distribution *without* redistributing to existing systems.
- Kevin Holman
  February 19, 2019 at 9:38 am
  
  Yuck. That’s exactly what my script does. There isnt a simple method to add one system to the distribution, so yes I get everyone that is existing, add the new system, then add all of them to the distribution. I did not realize this reset the health state of all monitors using a RunAs account. I’d have to test.
  
  We need a method to add systems individually to an existing distribution to resolve this – assuming that does not cause a health state impact on existing systems. A customer would need to raise a support case filing this as a bug/feature request.
- Kevin Holman
  February 19, 2019 at 9:38 am
  
  By the way – this makes me like using Healthservice SID’s as the solution, that much more.
  - John Austin
    February 19, 2019 at 2:15 pm
    
    Thanks for the reply!
    
    Understood…I hadn’t considered filing it in that way, assuming I had always just done something wrong or missed something in the script. It’s worth trying.
    
    (I would like the SID solution as well, if we could pull it off.)
    
    Thanks!
John Austin
April 1, 2019 at 2:57 pm

Follow up: You can use PowerShell to add to the distribution instead of wiping/rewriting the distribution every time. (Thanks to Kevin Justin for chasing this down for me.) Here’s a short, simplified sample:

# SCOM Connection
$MG = New-Object Microsoft.EnterpriseManagement.ManagementGroup(“localhost”)
# Create a new List to add to
$List = New-Object ‘System.Collections.Generic.List[Microsoft.EnterpriseManagement.Monitoring.MonitoringObject]’
#Get a list of Health Service instances
$instances = Get-SCOMClass -DisplayName “Health Service” | Get-SCOMClassInstance
#Get the HS instance you care about
$healthservices = $instances | where {$_.DisplayName -eq ‘server.name.here’}
#Add that HS instance to your list
$List.Add($healthservices)
#Get the RunAs Account
$account = ($MG.GetMonitoringSecureData() | where {$_.Name -eq “RunAsAccountNameHere”})
#Add the list of instances to the account distribution
$mg.Security.ApproveCredentialForDistribution($account, $List)

Automating Run As Account Distribution – Finally!

5 Comments

Leave a Reply Cancel reply