Something that a fellow PFE (Brian Barrington) called to my attention, with SCOM 2016 agents, when installed on a Domain Controller: the agent just sits there and does not communicate.
The reason? Local System is denied by HSLOCKDOWN.
HSLockdown is a tool that grants or denies a particular RunAs account access to the SCOM agent Healthservice. It is documented here.
When we deploy a SCOM 2016 agent to a domain controller – you might see it goes into a heartbeat failed state immediately, and on the agent – you might see the following events in the OperationsManager log:
Log Name: Operations Manager
Source: HealthService
Event ID: 7017
Task Category: Health Service
Level: Error
Computer: DC1.opsmgr.net
Description:
The health service blocked access to the windows credential NT AUTHORITY\SYSTEM because it is not authorized on management group SCOM. You can run the HSLockdown tool to change which credentials are authorized.
Followed eventually by a BUNCH of this:
Log Name: Operations Manager
Source: HealthService
Event ID: 1102
Task Category: Health Service
Level: Error
Computer: DC1.opsmgr.net
Description:
Rule/Monitor “Microsoft.SystemCenter.WMIService.ServiceMonitor” running for instance “DC1.opsmgr.net” with id:”{00A920EF-0147-3FCC-A5DC-CEC1CA93AFED}” cannot be initialized and will not be loaded. Management group “SCOM”
If you open an Elevated command prompt, and browse to the SCOM agent folder – you can run HSLOCKDOWN /L to list the configuration:
There it is. NT Authority\SYSTEM is denied.
I’ll be researching why this change was made – this did not happen by default in SCOM 2012R2.
In the meantime – the resolution is simple.
On domain controllers – simply run the following command in the agent path where HSLOCKDOWN.EXE exists:
HSLockdown.exe /A “NT AUTHORITY\SYSTEM”
This will remove the explicit deny for Local System, and add it to the allowed list.
Here is an example (my management group name is “SCOM”)
Restart the SCOM Microsoft Monitoring Agent Service (Healthservice) after you make this change for it to take effect.
I’ve got a couple of questions about this, if you don’t mind.
We’re deploying this using the least-priv settings as outlined in the MP guide for the AD DS MPs (2012 & 2016 – using a domain account).
It seems one thing we can’t do is monitor services. Setting this all up is fairly straightforward, I’m curious as to why are seeing these errors for various services. The only real thing that seems wonky in the guide is that it say to make the account a member of the “Local Users” group – which of course doesn’t exist on a DC. We have confirmed that this service account can log in to DCs interactively, and all the other permissions in the MP guide are configured:
Error getting state of service
Service: NTDS
Error: 0x80070005
Details: Access is denied.
One or more workflows were affected by this.
Workflow name: Microsoft.Windows.Server.2012.R2.AD.AvailabilityEssentialService.NTDS.ServiceCheck
Instance name:
Instance ID: {89FB8096-AA51-235E-429D-2F61161A9FD1}
Management group:
I typically just advise using Local System for monitoring, so I rarely see customers use low priv on DC’s…. but…. these service monitors just use the standard Windows!Microsoft.Windows.CheckNTServiceStateMonitorType which is the most common service monitor type used from the Windows Library, so nothing special is needed. If you are getting access denied, you dont have high enough priviledge to query WMI for service status.
https://serverfault.com/questions/556138/permissions-required-for-a-local-user-to-query-services
This solution worked on all of my 2016 DCs but not my one single 2012 DC (which I can’t upgrade at the current time). When I run the exact same command (HSLockdown.exe /A “NT AUTHORITY\SYSTEM”) that worked on all of the 2016 DCs, it laughs at me saying”
“Accounts must be specified in one of the following fully-qualified formats….”
Hello SCOTT,
Try adding your management group name in command.
eg if my management grup name is xyz then, below command:
HSLockdown.exe “Managem” /A “NT AUTHORITY\SYSTEM”
I know I am little late to this thread, but this was very helpful for me. As for Scott’s issue, I ran into the same thing, what worked for me was removing NT AUTHORITY\SYSTEM from the denied list first.
Hi Kevin,
In SCOM 2019 as well we need to make this changes manually. Its not added by default in SCOM 2019.
Regards, Vipin
Can anyone please explain why it says NT Authority\System has no access. Didn’t we configure a run as account? So why is the error not reporting this action account?
Hi Kevin,
When I ran HSLockdown.exe /L ,I even not found “NT AUTHORITY\SYSTEM in the allowed list not is denied list…
Hi Kevin,
getting event ID 11052 (Module was unable to convert parameter to a double value) on SCOM agent windows server, due which unable to fetch utilization report is generated blank.
please provide solution for to fix the same.