KB Article for OpsMgr
List of New Features
Download Update Rollup from the Catalog
Download the NEW Simplified Management Server Update EXE
Download Updated UNIX/Linux Management Packs
Recommended hotfix page
New Features:
- Discover Azure Monitor SCOM Managed Instance (preview) from Operations Manager console
- Telemetry capability enhancements for Notification channels and subscriptions
- Added support for Rocky Linux 8, Alma Linux 8, and systems with OpenSSL 3.0: RHEL 9.0 and Ubuntu 22
Key fixes:
- Fixed an issue where editing an existing Maintenance Mode schedule does not change the Reason and/or Comment.
- Fixed an issue where when setting Maintenance Mode via PowerShell, the Availability Reports were not reflecting correct information.
- Fixed the issue in which Member column in group View was introducing delay in group operations.
- Fixed an issue of users getting HTTP200 error when trying to setup Log Analytics connection.
- The script (GetOpsMgrDBPercentFreeSpace.vbs) which is part of System Center Core Monitoring MP monitor has been moved from VBS to PowerShell and, now reports Operations Database free space correctly.
- A new registry key (for debugging purposes) to enable Bad.xml file creation is introduced in UR5 which does not exist by default but needs to be created
- Fixed multiple Web Console Security Vulnerabilities.
- The organization of temporary files used for Kerberos based authentication is further enhanced to prevent any misuse.
- Linux – Fixed Data parsing issues in Linux agent that might cause the agent to crash.
- Linux – Fixed an issue where msgAuthenticationParameters needs to have 0 length during engine discovery of SNMPv3 devices.
- Linux – Fixed an issue related to SNMP Discovery where we see MonitoringHost.exe crashes.
- Linux – Fixed the issue where user was unable to run Get-SCXAgent and Invoke-SCXDiscovery remotely using Invoke-Command.
- Linux – Fixed Linux agent crash issue caused by variable out of scope issue for _HandleGetClassReq.
- Linux – Fixed an issue that would causes Linux agent to crash when DSC provider is installed.
NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2019 and never applied an update rollup – you can go straight to the latest one available.
Let’s get started
From reading the KB article – the order of operations is:
- Install the update rollup package on the following server infrastructure:
- Management Servers
- Audit Collection Servers
- Web Console Servers
- Gateway Servers
- Operations Console Servers
- Reporting Server
- Apply Agent Updates
- Update Unix/Linux MP’s and Agents
Management Servers
Updates in SCOM 2019 have CHANGED. There is a new process for updating management servers that differs from previous versions of SCOM. Download the single file management server update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates.
- Download: HERE
It doesn’t matter which management server I start with. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I have multiple management servers My first two management servers hold 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.
Notice the new EXE file, and a MSP file exist for the Server update. The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching. I will ONLY demonstrate and recommend the EXE file for the Management Server role update.
Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.
- ***Note: One of the changes in SCOM 2019 Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL. The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases. You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.
My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.
I will start with KB5025123-AMD64-Server.exe
This is a self-extracting executable, that kicks off a simple update tool. Accept the license terms, and click “Install”
This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.
NOTE: In my case, my SCOM Management server IMMEDIATELY rebooted without pause or prompt.
If you have an issue – you can review the setup logs:
- Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
- Patch log: C:\Users\<UserName>\Appdata\Local\Temp
- SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
- MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log
Web Console Update – since this management server also runs a SCOM Web Console, I will run: KB5025123-AMD64-WebConsole.msp
Console Update – (make sure your console is closed): KB5025123-AMD64-Console.msp
You can reboot the server at this time if you were prompted to in order to complete the update. If you were not prompted to, you do not need to.
Additional Management Servers:
Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers. You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.
You can use the same EXE file and MSP files (where applicable) you used for the first management server. The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if they are not needed.
ACS Update: (Audit Collection Services)
One of my management servers is also my ACS Audit Collection Server role. I will apply the update for that: KB5025123-AMD64-ACS.msp
Updating Gateways:
Open an elevated command prompt, and run the update: KB5025123-AMD64-Gateway.msp
The update launches a UI and quickly finishes.
Updating Reporting:
On your server that hosts the SCOM Reporting role, run the update: KB5025123-AMD64-Reporting.msp
Note: This prompted for a reboot on my reporting server, so plan accordingly.
Update Agents:
Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):
***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin. The Agents MUST have “Remotely Manageable” set to “Yes”.
You can approve these – which will result in a success or failure message once complete:
Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:
And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery. This has long been a pain point in SCOM:
https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
Update UNIX/Linux MPs and Agents:
You can get the current Unix/Linux MP updates HERE.
Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor. Here is the FULL list:
In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update. Yours may vary, depending on what previous versions you are upgrading from:
These can take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.
Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits. If they are not present or not updated after import, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.
After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:
Now you can deploy the Linux agent updates:
If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:
Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.
Verifying the update
There are new views in the SCOM console to help with this and make this process MUCH easier. You do need to wait long enough for the discoveries to run in order for these to update the views.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known Issues:
1. Health Explorer and other web console features that use popups no longer work.
- Make a backup copy of \Program Files\Microsoft System Center\Operations Manager\WebConsole\Dashboard\main.js
- Edit main.js (using notepad is fine).
- Search for the string “sandbox” and on the THIRD hit, edit the string immediately after the third “sandbox” adding: allow-popups allow-forms
- See image example below:
- Save the file, then clear the browser cache AND restart IIS service (iisreset).
2. The new OperationsManager DB free space script requires additional permissions.
- This script was recently updated from VBS to PowerShell, and some additional instance level permissions are required for the SQL server hosting the OperationsManager database
- If you are missing these permissions, you will see the following event on one of your management servers:
- Log Name: Operations Manager
Source: Health Service Script
Event ID: 100
Level: Warning
Description:
GetOpsMgrDBPercentFreeSpace.ps1 : Exception calling “Fill” with “1” argument(s): “VIEW SERVER STATE permission
was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.
VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
The user does not have permission to perform this action.”
At line:234 char:5
+ $adp.Fill($dt) | out-null
+ ~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : SqlException - To resolve this, you must grant the “VIEW ANY DEFINITION” and “VIEW SERVER STATE” SQL instance level permission to the Management Server Action account SQL login on the SQL instance hosting the OperationsManager database. If you use AlwaysOn, make sure you set this permission all replica servers.
- Open the login properties, select “Securables”, and add “View any definition” and “View server state” permissions:
3. You MUST check for duplicate aliases before applying UR5 to SCOM 2019 UR3 or previous versions. If you have any MP’s with duplicate aliases the SDK will throw errors after the upgrade
- In previous versions of SCOM, duplicate aliases (with different case) was allowed. This is no longer allowed starting with SCOM 2019 UR4. Please run the PowerShell or SQL scripts below to check for this condition and resolve this if you are upgrading from SCOM 2019 UR3 or previous.
- If impacted, your SCOM console might be stuck while “connecting”.
- SDK PowerShell commands might return an error with “An object of class ManagementPackClass with ID <GUID> was not found”
- Event 27000 In the OpsManager event log : “ An Item with the same key has already been added”
- Failed setup/upgrade installation with messages in the log:” An Item with the same key has already been added”
Example of a bad MP:
To detect MPs with this issue, you can use the following PowerShell or SQL query. After detection, you will need to export these MP’s and re-label one of the duplicate Aliases, and re-label it anywhere it was used in the body of the XML.
############################################ #Identify MPs imported with duplicate Aliases $mps = Get-SCOMManagementPack foreach ($mp in $mps) { $hashTable = @{} foreach ($ref in $mp.References) { try {$hashTable.Add($ref.Key, $ref.Value)} catch { $MPName = $mp.Name $MPDisplayName = $mp.DisplayName $MPVersion = $mp.Version "MP contains duplicate aliases: Name=($MPName) DiplayName=($MPDisplayName) Version=($MPVersion)" } } } ############################################
SQL query:
-- LIST ALL MPs that have a duplicate Alias reference declare @mpFriendlyName nvarchar(255), @mpName nvarchar(255), @mpId uniqueidentifier, @mpXml as xml create table #badMPTable ( mpId uniqueidentifier, mpName nvarchar(255), mpFriendlyName nvarchar(255) ) declare mp_cursor cursor local forward_only read_only for select MPFriendlyName, MPName, ManagementPackId, convert(xml, MPXML) from ManagementPack open mp_cursor fetch next from mp_cursor into @mpFriendlyName, @mpName, @mpId, @mpXml while @@FETCH_STATUS = 0 begin select n.value('@Alias','nvarchar(255)') as mpRef into #tempRefs from @mpXml.nodes('/ManagementPack/Manifest/References/Reference') as a(n) if exists( select count(*) from #tempRefs group by mpRef having count(*) > 1 ) begin insert into #badMPTable ( mpId, mpName, mpFriendlyName ) values ( @mpId, @mpName, @mpFriendlyName ) end drop table #tempRefs fetch next from mp_cursor into @mpFriendlyName, @mpName, @mpId, @mpXml end close mp_cursor deallocate mp_cursor select * from #badMPTable drop table #badMPTable --End
4. Once you apply an update rollup, you cannot install, or re-install SCOM WEB CONSOLE, or SCOM REPORTING roles.
- You will see an error in the setup UI (when you supply a management server name) that states “Unable to connect to the Data Access service for this management server. Ensure the Data Access service is running and that the service, the management group, and setup are all the same version.”
- In the setup log, located at C:\Users\<username>\AppData\Local\SCOM\LOGS\OpsMgrSetupWizard.log – the last line will appear similar to: Error:The management server is a different version than the current setup build. Please use a different management server or the correct version of setup. Server Version: 10.19.10505.0
- SCOM Reporting Setup looks for a very specific value in the SQL database, which has been updated by the Update Rollup and setup is now blocked.
- Apply the following workaround to install/reinstall SCOM WebConsole or Reporting role:
- QUERY the OPERATIONSMANAGER database, and record the VERSION number that is returned. You will need this value later. You need to change the PrincipalName to your SCOM Management server that you point the reporting install to.
-- 10.19.10050.0 - 2019 RTM -- 10.19.10311.0 - 2019 UR1 -- 10.19.10349.0 - 2019 UR1 with post UR1 Hotfix -- 10.19.10407.0 - 2019 UR2 -- 10.19.10505.0 – 2019 UR3 -- 10.19.10569.0 – 2019 UR4 -- 10.19.10606.0 – 2019 UR5 USE OperationsManager SELECT PrincipalName, Version FROM MTV_HealthService WHERE IsManagementServer = 1 AND PrincipalName = 'OMMS1.opsmgr.net'
UPDATE the VERSION entry in the OpsDB to match the RTM version number which is 10.19.10050.0 just for this management server.
UPDATE MTV_HealthService SET Version = '10.19.10050.0' -- 2019 RTM WHERE PrincipalName = 'OMMS1.opsmgr.net'
Install SCOM 2019 Reporting, and choose this same Management Server. Reporting install will work now.
REVERT the VERSION entry in the OpsDB to match the original value you recorded
UPDATE MTV_HealthService SET Version = '10.19.10606.0' -- 2019 UR5 WHERE PrincipalName = 'OMMS1.opsmgr.net'
5. When you run the KB5025123-AMD64-Server update, sometimes the tasks fail to update the SQL databases, or perform the Agent Pending Management task.
This can happen when your permissions are not set correctly for your RunAs accounts. There are 3 tasks that get deployed from the Management Pack included in the Update Rollup:
MP: Microsoft.SystemCenter.DBUpdateTask
- Database update task
- Datawarehouse update task
- Agent pending management task
The “Database update task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Datawarehouse update task” runs as whatever action account is associated with the RunAs Profile: “Data Warehouse Account”. By default the RunAs account “Data Warehouse Action Account” is associated with this profile, so the task will execute as the credential configured in the “Data Warehouse Action Account” RunAs account, which should be the Data Warehouse Write Action account you specified when you installed SCOM. This credential has a very high privilege to the OperationsManagerDW database (db_owner) so it can modify anything necessary. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Agent pending management task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
If any of these fail, it is likely that someone has modified the default permissions for your action accounts to the SQL databases, or someone has incorrectly modified the default RunAs profile associations. Please review your permissions against: SCOM 2019 Security Account Matrix – Kevin Holman’s Blog
You can always just run the SQL script update tasks manually, by using the method shown in previous versions of SCOM as seen here: UR10 for SCOM 2016 – Step by Step – Kevin Holman’s Blog
Done!
Thanks alot for the good instructions on how to Update it the new way. Really helps alot. I just recently updated to UR 4, in hope to get RHEL 9.1 to work. Turns out, it doesn’t really support 9 at all, only 9.0 with a hotfix which isn’t available anymore on 9.1. Is this Update gonna fix that issue or is that still a work in progress on Microsofts end?
Nevermind, there was an update for rhel recently for openssh Version 8.7p1-29. Updating that solved my issue, in case anyone has ssh errors aswell.
Hi Kevin,
Once SCOM Agent sends Alert information to the SCOM Management server, what is the alert packet size in KB?
we would like to know how much bandwidth will it use?
Thanks
Atif
Cheers Kevin,
Hmmm, strange that the KB article would include the following passage:
“The prerequisites required to install this update is Update Rollup 4 for System Center 2019 Operations Manager.”
Documentation error on their part, or is UR4 truly a prerequisite to UR5 (maybe due to the duplicate MP Alias issue)?
Thanks,
UR4 is NOT a prereq. The KB just calls that out because if you are upgrading from anything previous to UR4 – you need to do a bad alias MP search first. This is called out in my known issues.
Hey Kevin
Is it possible for you to review your post about the KMS-Management Pack? I added a comment about a week ago.
We’re experiencing problems with the Idle Minutes Monitor. It’s triggering when it should not. I’ve tried playing around with overriding the thresholds but nothing worked. The Monitor is definitely falsely triggering. Could you maybe take a look at this?
Kind regards
Manuel
After updating the Web Console role to UR5, opening the health explorer does nothing. In the developer options console, I have found the following entry:
“Blocked openingin a new windows because the request was made in a sandboxed frame whose ‘allow-popups’ permission is not set.”
Has anyone encountered this?
As of now, this issue has been described as the first of the known issues. I successfully applied the workaround described by Kevin. After applying the workaround (including IISreset), you may also need to clear your browser’s cache.
We’re still investigating. We’ve had SCOM2019 components updated using WSUS. We were unaware this may or may not be supported (anymore?).
The datawarehouse DB now seems damaged, jobs look weird, autorisations missing and a specific alert coming back
Error 31555 in eventlog
Data Warehouse maintenance mode synchronization process failed to read data
Health Explorer and other web console features that use popups no longer work.
When I review my “\Program Files\Microsoft System Center\Operations Manager\WebConsole\Dashboard\main.js”, I only get 2 hits on “sandbox”. The issue is manifesting in our Web Console (inability to open things like Health Explorer).
Thoughts?
Same problem here…
Anyone have a solution?
I get only 2 hits on “sandbox” just like Scott
I get only 2 hits on “sandbox” just like Scott
I have the same challenge. Maybe Kevin has some input here?
Any update on this? I only get two hits myself.
I got it to work. Search for u0275eld. Find this line:
i[“\u0275eld”](0,0,null,null,0,”iframe”,[[“frameBorder”,”0″],[“id”,”frame”],[“role”,”presentation”],[“sandbox”,”allow-scripts allow-same-origin”]
Change to:
i[“\u0275eld”](0,0,null,null,0,”iframe”,[[“frameBorder”,”0″],[“id”,”frame”],[“role”,”presentation”],[“sandbox”,”allow-popups allow-forms allow-scripts allow-same-origin”]
Do the iisreset and its working for me.
I only have “2 hits” on sandbox as well, I can verify the above approach works!
I can confirm that Mike’s solution worked for me. I was able to put machines in maintenance mode via the web console.
i am receiving error in Database configuration :roduct: System Center Operations Manager Server – Update ‘System Center 2019 Operations Manager Update Rollup 5 Patch’ could not be installed. Error code 1603
I reviewed the permissions and all looks good the dataware house write account has the highest permission as well as the SCOM action account
I am running scom 2019 UR4 with gMSA service accounts. Does UR5 compitable with gMSA?
After applying the UR5 on our management servers in the All Management Servers Resource Pool the Resource Pool broke. All SCOM HealthService Watchers are grey and haven’t found a solution yet. So maybe anyone or Kevin got an idea what’s up with my management servers? All are green in Administration but HealthService Watchers grey.
Morning, I am currently seeing this issue in SCOM 2019 UR5. Using Web console, I attempt to put a server into maintenance mode. I can select everything fine, but when clicking OK, the box just sits there, it does not close nor does it lockup. I can select Cancel and the box will close.
I have a QC/Test tier and a Prod tier, both have the same issue.
I have tried Edge and Chrome, same result, as well as using an Incognito Window.
Anyone else seeing this?
Thank you,
Tony
I have exactly the same issue, Any solution to this ?
Tony, look above to my post from 8/24. I fixed it by doing that.
After installing UR5 in our environment we got an error while trying to update our Linux agents. The error message is “Failed to find a matching agent kit to install”. I found out that Microsoft released a hotfix for this issue, read about it here
https://learn.microsoft.com/en-us/answers/questions/1342979/scom-2022-getting-failed-to-find-a-matching-agent and here
https://support.microsoft.com/en-au/topic/system-center-operations-manager-now-uses-compiler-mitigated-mps-and-packages-for-monitoring-linux-unix-kb-5028684-b94779fe-1c05-47fe-9066-c7470f0d130a.
The hotfix is available for the management server and the console. I was able to install the hotfix on our primary MS, on the secondary it failed with error 1603, there is no Logfile or Eventlogmessage to tell me what the exact issue is 🙁 The hotfix for the console was successful on both MS. After installing the hotfix the update of the agents still fails with the error “sudo: no tty present and no askpass program specified”. We are still investigating.
@Kevin: Maybe you can put this in the Known Issues section?”
Agreed – will do.
I would also appreciate an update with this info. We now need to apply KB5028684, due to the “Failed to find a matching agent kit to install” error when upgrading Linux agents from the console.
All works fine, but no new file in DownloadedKits folder. Even if I upgraded the MPs and restarted the monitoring services. Any idea ?
We had to wait 24 hours for the new agents to show up after we imported the Linux MPs and restarted the services.
Fixed the issue where new file were not copied in folder DownloadedKits. But now, upgrading the agent, I have this error:
Failed to find a matching agent kit to install.
?????
Maintenance mode in Web Console not working — I have found that after applying UR5 and KB5028684, I am not able to put machines in maintenance mode through the web console. It presents the maintenance mode dialog, however it does not respond to the OK button. Is anyone else having this issue?
Installation a newest management packs and hotfix on MS’s and Console’s received discovery issue “Failed to find a matching agent kit to install.”
https://support.microsoft.com/en-us/topic/system-center-operations-manager-now-uses-compiler-mitigated-mps-and-packages-for-monitoring-linux-unix-kb-5028684-b94779fe-1c05-47fe-9066-c7470f0d130a
Hello,
after upgrading to UR5 I get an error message when trying to update the Management Packs online or accessing the Updates and Recommendations page.
I get:
“Operations Manager cannot connect to the Web Service
The Microsoft Management Pack Catalog Web Service cannot be contacted at this time. There is either a problem with your Internet connection or the Web service is offline.”
I already asked Google, but that was not really helpful. THe Management Server has full access to the internet and nothing is blocked. I get the message also from my client. Do you have any idea, why this happens?
I don’t know what to do at the moment to fix this.
Thanks!
Regards
Chris
Do you have a doc for UR6 (even though it’s substantially the same)?
I’d really love that my self aswell, I hope there will be a guide soon 🙂
UR6 doc ?
Kevin, I’d say add that a check for SQL broker being enabled should be done before upgrading to UR5. It’s probably more common than we think 🙂