KB Article for OpsMgr
Download Update Rollup from the Catalog
Download the NEW Simplified Management Server Update EXE
Download Updated UNIX/Linux Management Packs
Recommended hotfix page
Key fixes:
- Fixed the failure of favorite reports in SCOM 2022 Web console with HttpParseException.
- Fixed Teams notifications issues in SCOM 2022 UR2.
- Resolved random crashes of SCOM console in global search with NullReferenceException.
- You can now add Dashboard to Workspace which was not possible previously with 2022 UR1.
- Dashboard Performance widget now works which was previously not working with 2022 UR1.
- Resolved the failure of workflows using WMIProbe.
- Unix/Linux computers view from SCOM 2022 Web console now opens properly.
- This Update Rollup includes the fixes in KB5033752 and KB5037360
NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2022 and never applied an update rollup – you can go straight to the latest one available.
Let’s get started
From reading the KB article – the order of operations is:
1. Install the update rollup package on the following server infrastructure:
- Management Servers
- Audit Collection Servers
- Web Console Servers
- Gateway Servers
- Operations Console Servers
- Reporting Server
2. Apply Agent Updates
3. Update Unix/Linux MP’s and Agents
Management Servers
There is a new process for updating management servers that differs from previous versions of SCOM. Download the single file management server EXE update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates with a UI to show you success/fail and progress.
It doesn’t matter which management server I start with. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I have multiple management servers My first two management servers hold 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.
Notice the EXE file, and a MSP file exist for the Server update. The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching. EITHER of these files will patch the server and attempt to run the SQL scripts and MP imports. The EXE simply provides a visual progress. That is the primary difference. I will ONLY demonstrate and recommend the EXE file for the Management Server role update.
Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.
- ***Note: One of the changes in SCOM 2019 and later Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL. The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases. You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.
My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.
I will start with KB5055455-amd64-Server.exe
This is a self-extracting executable, that kicks off a simple update tool. Accept the license terms, and click “Install”
This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.
You SHOULD see something like the following when complete.
Unfortunately – the SCOM product group has not resolved an issue I have reported for years – that sometimes this EXE update forces an immediate reboot of the management server as soon as this update completes, and you do not get to see if all steps were successful. That’s exactly what happened to me when testing this update, so plan accordingly.
If you have an issue – you can review the setup logs:
- Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
- SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
- MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log
Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update: KB5055455-amd64-WebConsole.msp
Next – install the Console Update (make sure your console is closed): KB5055455-amd64-Console.msp
You can reboot the server at this time if you were prompted to in order to complete the update. If you were not prompted to, you do not need to.
Additional Management Servers:
Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers. You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.
You can use the same EXE file and MSP files (where applicable) you used for the first management server. The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if they are not needed.
Unfortunately – the SCOM product group has not resolved an issue I have reported for years – that sometimes this EXE update forces an immediate reboot of the management server as soon as this update completes, and you do not get to see if all steps were successful. That’s exactly what happened to me when testing this update, so plan accordingly.
ACS Update: (Audit Collection Services)
One of my management servers is also my ACS Audit Collection Server role. I will apply the update for that:
KB5055455-amd64-ACS.msp
Updating Gateways:
Open an elevated command prompt, and run the update: KB5055455-amd64-Gateway.msp
The update launches a UI and quickly finishes.
Updating Reporting:
On your server that hosts the SCOM Reporting role, run the update: KB5055455-amd64-Reporting.msp
Update Agents:
There is a issue in SCOM 2022 RTM that you need to fix first before pushing agent updates.
Browse to your \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory and delete the following two files if they exist:
- KB3117586-amd64-Agent.msp
- KB9999999-amd64-Agent.msp
This appears to be a bug in SCOM 2022 RTM, as these files should not exist.
You may also delete any previous UR files, that are left behind from previous Update Rollups.
Once these files are deleted – you can continue.
Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):
***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin. The Agents MUST have “Remotely Manageable” set to “Yes”.
You can approve these – which will result in a success or failure message once complete:
Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:
And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery. This has long been a pain point in SCOM:
https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
Update UNIX/Linux MPs and Agents:
You can get the current Unix/Linux MP updates HERE.
Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor. Here is the FULL list:
In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update. Yours may vary, depending on what previous versions you are upgrading from:
Uh oh! Got an error on one of the MP’s:
UNIX/Linux Shell Command and Script Library could not be imported.
If any management packs in the Import list are dependent on this management pack, the installation of the dependent management packs will fail.
Cannot find resource with ID Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Authoring.ShellCommandTemplate, Microsoft.SystemCenter.CrossPlatform.UI.CL.Authoring.ShellCommandTemplate, Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Authoring.ScriptTemplate, Microsoft.SystemCenter.CrossPlatform.UI.CL.Authoring.ScriptTemplate, Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.Authoring.Common, Microsoft.SystemCenter.CrossPlatform.UI.CL.Authoring.Common.
There is a BUG in the Linux MP downloads, in that the files contain both a .mp file and a .mpb file for Microsoft.Unix.ShellCommand.Library:
To work around this – ONLY import Microsoft.Unix.ShellCommand.Library.mpb
It will import fine as long as you do not select the Microsoft.Unix.ShellCommand.Library.mp file for import.
These can take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.
Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits. If they are not present or not updated after import, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.
After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:
Now you can deploy the Linux agent updates:
FAILED!
I got a failure with the following output:
“The SSL certificate contains a common name (CN) that does not match the hostname”
I logged into a session with the Linux server, and checked the cert:
openssl x509 -noout -in /etc/opt/microsoft/scx/ssl/scx.pem -subject -issuer -dates
Which resulted in:
subject=CN = ubuntu20, CN = ubuntu20 issuer=CN = SCX-Certificate, title = SCX55a8126f-c88a-4701-9754-8625324426ff, DC = OM2
This is wrong, but I am not sure what changed it. The DNS name is missing. So to resolve this I run:
sudo /opt/microsoft/scx/bin/tools/scxsslconfig -f -h ubuntu20 -d opsmgr.net
Now when I check the cert I get:
subject=DC = net, DC = opsmgr, CN = ubuntu20, CN = ubuntu20.opsmgr.net issuer=DC = net, DC = opsmgr, CN = ubuntu20, CN = ubuntu20.opsmgr.net
At this point, I re-run the agent upgrade, which will re-sign the cert:
A quick check of the certificate again reveals:
subject=DC = net, DC = opsmgr, CN = ubuntu20, CN = ubuntu20.opsmgr.net issuer=CN = SCX-Certificate, title = SCX55a8126f-c88a-4701-9754-8625324426ff, DC = OM1
Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.
Verifying the update
There are new views in the SCOM console to help with this and make this process MUCH easier. You do need to wait long enough for the discoveries to run in order for these to update the views.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known Issues:
1. SCOM 2022 RTM has a bug that left two agent update files in \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory.
You need to delete the following two files (and any other older KB update files if you like):
- KB3117586-amd64-Agent.msp
- KB9999999-amd64-Agent.msp
2. The messages about Data Warehouse errors contain Chinese characters instead of English
There is a bug in SCOM 2022 RTM that has never been fixed. When you install SCOM, there are sysmessages added to the master database for the SQL instance. For the Data Warehouse SQL server, these sysmessages contain chinese characters by mistake.
To correct this issue, please download and execute the SQL scripts at: SCOM 2016, 2019 and 2022: Event 18054 errors in the SQL application log – Kevin Holman’s Blog
3. The new OperationsManager DB free space script requires additional permissions.
- This script was updated in UR1 from VBS to PowerShell, and some additional instance level permissions are required for the SQL server hosting the OperationsManager database
- If you are missing these permissions, you will see the following event on one of your management servers:
- Log Name: Operations Manager
Source: Health Service Script
Event ID: 100
Level: Warning
Description:
GetOpsMgrDBPercentFreeSpace.ps1 : Exception calling “Fill” with “1” argument(s): “VIEW SERVER STATE permission
was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.”
At line:234 char:5
+ $adp.Fill($dt) | out-null
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SqlException - To resolve this, you must grant the “VIEW ANY DEFINITION and “VIEW SERVER STATE” SQL instance level permission to the Management Server Action account SQL login on the SQL instance hosting the OperationsManager database. If you use AlwaysOn, make sure you set this permission all replica servers.
- Open the login properties, select “Securables”, and add “View any definition” and “View Server State” permissions:
4. When you run the Management server role update, sometimes the tasks fail to update the SQL databases, or perform the Agent Pending Management task.
This can happen when your permissions are not set correctly for your RunAs accounts. There are 3 tasks that get deployed from the Management Pack included in the Update Rollup:
MP: Microsoft.SystemCenter.DBUpdateTask
- Database update task
- Datawarehouse update task
- Agent pending management task
The “Database update task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Datawarehouse update task” runs as whatever action account is associated with the RunAs Profile: “Data Warehouse Account”. By default the RunAs account “Data Warehouse Action Account” is associated with this profile, so the task will execute as the credential configured in the “Data Warehouse Action Account” RunAs account, which should be the Data Warehouse Write Action account you specified when you installed SCOM. This credential has a very high privilege to the OperationsManagerDW database (db_owner) so it can modify anything necessary. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Agent pending management task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
If any of these fail, it is likely that someone has modified the default permissions for your action accounts to the SQL databases, or someone has incorrectly modified the default RunAs profile associations. Please review your permissions against: SCOM 2022 Security Matrix
Done!
