Menu Close

UR2 for SCOM 2022 – Step by Step

image

KB Article for OpsMgr

Download Update Rollup from the Catalog

Download the NEW Simplified Management Server Update EXE

Download Updated UNIX/Linux Management Packs

Recommended hotfix page

 

New Features:

  • Discover Azure Migrate, generate Business Case and Assessment through SCOM Console

Key fixes:

  • See the KB article.  There are too many fixes to list here.

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2022 and never applied an update rollup – you can go straight to the latest one available.

IMPORTANT:  UR2 was pulled shortly after release due to an issue with the DataWarehouse SQL script failing 100% of the time.  UR2 was repackaged with an updated version number, so if you downloaded the early UR2, you should re-download the current version that is fixed.  The interim version that was pulled is 10.22.10605.0 and the fixed and current version is 10.22.10610.0

 

 
Let’s get started

From reading the KB article – the order of operations is:

1. Install the update rollup package on the following server infrastructure:

  • Management Servers
  • Audit Collection Servers
  • Web Console Servers
  • Gateway Servers
  • Operations Console Servers
  • Reporting Server

2. Apply Agent Updates

3. Update Unix/Linux MP’s and Agents

 

Management Servers

image

There is a new process for updating management servers that differs from previous versions of SCOM.  Download the single file management server EXE update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates with a UI to show you success/fail and progress.

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I have multiple management servers  My first two management servers hold 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

Notice the EXE file, and a MSP file exist for the Server update.  The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching.  EITHER of these files will patch the server and attempt to run the SQL scripts and MP imports.  The EXE simply provides a visual progress.  That is the primary difference.  I will ONLY demonstrate and recommend the EXE file for the Management Server role update.

Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.

  • ***Note:  One of the changes in SCOM 2019 and later Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL.  The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases.  You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.

My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.

I will start with KB5031649-amd64-Server.exe

This is a self-extracting executable, that kicks off a simple update tool.  Accept the license terms, and click “Install

This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.

image

If you have an issue – you can review the setup logs:

  • Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
  • SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
  • MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log

Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update: KB5031649-amd64-WebConsole.msp

Next – install the Console Update (make sure your console is closed):   KB5031649-amd64-Console.msp

You can reboot the server at this time if you were prompted to in order to complete the update.  If you were not prompted to, you do not need to.

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

You can use the same EXE file and MSP files (where applicable) you used for the first management server.  The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if they are not needed.

I used the EXE, and UR2 forced an immediate reboot of the server when the update was completed.  No prompts – it just rebooted.  I have seen this behavior before in previous UR2 and have reported this behavior to the product group several times.

 

ACS Update: (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

KB5031649-amd64-ACS.msp

image

 

Updating Gateways:

image

Open an elevated command prompt, and run the update:   KB5031649-amd64-Gateway.msp

The update launches a UI and quickly finishes.

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:   KB5031649-amd64-Reporting.msp

 

Update Agents:

image

image There is a issue in SCOM 2022 RTM that you need to fix first before pushing agent updates.

Browse to your \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory and delete the following two files if they exist:

  • KB3117586-amd64-Agent.msp
  • KB9999999-amd64-Agent.msp

This appears to be a bug in SCOM 2022 RTM, as these files should not exist.

image

You may also delete any previous UR files, that are left behind from previous Update Rollups.

Once these files are deleted – you can continue.

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin.  The Agents MUST have “Remotely Manageable” set to “Yes”.

image

You can approve these – which will result in a success or failure message once complete:

image

Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:

image

And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

 

Update UNIX/Linux MPs and Agents:

Linux - Wikipedia

You can get the current Unix/Linux MP updates HERE

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

image

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update.  Yours may vary, depending on what previous versions you are upgrading from:

image

These can take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits.   If they are not present or not updated after import, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.

After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:

image

Now you can deploy the Linux agent updates:

image

 

Uh oh….  I got the dreaded – Failed to find a matching agent kit to install

image

 

This issue is resolved with a POST UR2 hotfix.  The hotfix is available here:

Hotfix for System Center Operations Manager 2022 UR2 – Microsoft Support

The hotfix is two files, one for the console and one for the server.  I applied the hotfix and this resolved the agent upgrade issue

image

 

 

Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version. 

 

 

Verifying the update

There are new views in the SCOM console to help with this and make this process MUCH easier.  You do need to wait long enough for the discoveries to run in order for these to update the views.

image

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update. 

 

Known Issues:

image

1.  Linux agents might not be able to be upgraded with “Failed to find a matching agent kit to install.” error

2.  Console crashes after applying the UR2 Console update

  • Due to a new addition in UR2 console for Azure Migrate, the console will crash if the customer removed all the UNIX/Linux MP’s from SCOM.  Reimporting the “UNIX/Linux Core Library” (Microsoft.Unix.Library) management pack will resolve this issue.

3.  Health Explorer and other web console features that use popups no longer work.

  • Make a backup copy of \Program Files\Microsoft System Center\Operations Manager\WebConsole\Dashboard\main.js
  • Edit main.js (using notepad is fine).
  • Search for the string “sandbox” and on the THIRD hit, edit the string immediately after the third “sandbox” adding:  allow-popups allow-forms
  • See image example below:

image

  • Save the file, then clear the browser cache AND restart IIS service (iisreset).

4.  SCOM 2022 RTM has a bug that left two agent update files in \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory.

You need to delete the following two files:

  • KB3117586-amd64-Agent.msp
  • KB9999999-amd64-Agent.msp

5.  The messages about Data Warehouse errors contain Chinese characters instead of English

There is a bug in SCOM 2022 RTM that has never been fixed.  When you install SCOM, there are sysmessages added to the master database for the SQL instance.  For the Data Warehouse SQL server, these sysmessages contain chinese characters by mistake. 

image

To correct this issue, please download and execute the SQL scripts at:  SCOM 2016, 2019 and 2022: Event 18054 errors in the SQL application log – Kevin Holman’s Blog

6.  The new OperationsManager DB free space script requires additional permissions.

  • This script was updated in UR1 from VBS to PowerShell, and some additional instance level permissions are required for the SQL server hosting the OperationsManager database
  • If you are missing these permissions, you will see the following event on one of your management servers:
  • Log Name:      Operations Manager
    Source:        Health Service Script
    Event ID:      100
    Level:         Warning
    Description:
    GetOpsMgrDBPercentFreeSpace.ps1 : Exception calling “Fill” with “1” argument(s): “VIEW SERVER STATE permission
    was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.”
    At line:234 char:5
    +     $adp.Fill($dt) | out-null
    +     ~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : SqlException
  • To resolve this, you must grant the “VIEW ANY DEFINITION and “VIEW SERVER STATE” SQL instance level permission to the Management Server Action account SQL login on the SQL instance hosting the OperationsManager database.  If you use AlwaysOn, make sure you set this permission all replica servers.
  • Open the login properties, select “Securables”, and add “View any definition” and “View Server State” permissions:

image

7.  When you run the Management server role update, sometimes the tasks fail to update the SQL databases, or perform the Agent Pending Management task.

This can happen when your permissions are not set correctly for your RunAs accounts.  There are 3 tasks that get deployed from the Management Pack included in the Update Rollup:

MP:  Microsoft.SystemCenter.DBUpdateTask

  • Database update task
  • Datawarehouse update task
  • Agent pending management task

The “Database update task” runs as whatever action account is associated with the RunAs Profile:  “Operational Database Account”.   By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account.  Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

The “Datawarehouse update task” runs as whatever action account is associated with the RunAs Profile: “Data Warehouse Account”.  By default the RunAs account “Data Warehouse Action Account” is associated with this profile, so the task will execute as the credential configured in the “Data Warehouse Action Account” RunAs account, which should be the Data Warehouse Write Action account you specified when you installed SCOM.  This credential has a very high privilege to the OperationsManagerDW database (db_owner) so it can modify anything necessary.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

The “Agent pending management task” runs as whatever action account is associated with the RunAs Profile:  “Operational Database Account”.   By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account.  Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

If any of these fail, it is likely that someone has modified the default permissions for your action accounts to the SQL databases, or someone has incorrectly modified the default RunAs profile associations.  Please review your permissions against:  SCOM 2022 Security Matrix

 

Done!

image

33 Comments

  1. Nave Razy

    Hey Kevin, thanks for sharing your note on SCOM 2022 UR2 installation. I followed your steps and it worked like a charm. You saved me a lot of time and hassle. You’re awesome!

    • Kevin Holman

      There will soon be a post UR2 hotfix to resolve this issue with agent upgrade failing due to “Failed to find a matching agent kit to install.”

  2. Hung

    My console on the 2 management servers (with UR2) crashed after startinng the console with UR2 on the management server.
    Same crash on the second management server.

    Does someone have the same problem?

      • Kevin Holman

        There was an issue in UR1, there was a post UR1 hotfix for that. However, UR2 included this update. Your version should be 10.22.10610.0

        Are you sure you applied the UR2 console update to the console?

        • Hung

          >Are you sure you applied the UR2 console update to the console?
          yes.

          On my production environment “Still have UR1″ The UR1 Console on their mangemnet servers work fine, when i connect it with the test environment (UR2)

          I made some additional tests i found out that, if i start the Console UR2 with an Operator User ( without Operations Manager Administrators rights”. There is no crash. It works fine

        • Hung

          >There was an issue in UR1, there was a post UR1 hotfix for that. However, UR2 included this update. Your version should be 10.22.10610.0

          the console has the version 10.22.10610.0.
          I have installed it on serveral Windows 10 PCs. The Console (UR2) always crashed when i used the User with Operations Manager Administrators rights

          • Kevin Holman

            I cannot repro this. I have installed the SCOM console with UR2 on Windows 10 and Windows 11 and cannot see any crashes. I suggest opening a support case with MSFT.

          • Kevin Holman

            The same way you revert from all SCOM updates – restore from backups. All servers and databases.

  3. RODOLPHE

    Hello Kevin, As soon as I apply the UR1 or the UR2 on my SCOM 2022 platform, all my web console performance views (such as CPU usage percent, Memory usage percent, Logical drive C: usage percent and network bandwidth percent…) are broken. A simple rollback to the RTM (after running the KB uninstall command line) is sufficient to rebuilt the views. do you also encounter the same issue ? do you have any idea of the way to resolve it ?

    • RODOLPHE

      Hello Kevin, I recently opened a case to Microsoft support for the web console performance views issue and today, I had some good news. They easily reproduced the issue and they are working on a fix. They also gave me a workaround to wait until the new fix definitively solve the bug. So, you apply the UR2 on your Web console servers and then you roll-back 5 files (in the “C:\Program Files\Microsoft System Center\Operations Manager\WebConsole\Dashboard” folder) to their RTM version :
      – main.js
      – polyfills.js
      – runtime.js
      – index.html
      – styles.css

      I did not test the workaround, I prefer waiting for a rock solid fix

  4. Philippe M.

    Hi there,

    Thanks Kevin of all post you’ve done and tips you’ve gave to us… It’s great

    Regarding OperationsManager DB requiring additional permissions action, there is not only VIEW SERVER STATE issue.
    I’ve had also VIEW SERVER PERFORMANCE STATE permission issue. I’ve just applied the same procedure for this specific view and that’s it.

  5. Nikolay

    I faced the same issue with the console as HUNG above. I have 4 management servers and did upgrad from UR1 to UR2. SCOM console crashed on all of them. In the Application log I can see an event for an issue with .NET
    Application: Microsoft.EnterpriseManagement.Monitoring.Console.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ArgumentOutOfRangeException

    Then the next event ID1000
    Faulting application name: Microsoft.EnterpriseManagement.Monitoring.Console.exe, version: 10.22.10610.0, time stamp: 0x65561c08
    Faulting module name: KERNELBASE.dll, version: 10.0.17763.4974, time stamp: 0x909c5ed9
    Exception code: 0xe0434352
    Fault offset: 0x00000000000349b9

    I uninstalled consoles as a feature, then installed it again and upgraded to UR1

  6. Rob

    Hi there,

    my installation stalls at the same point as on UR2 for SCOM2019: Removing Backup files.
    Is there anything to do about this?
    At the .exe installation it stalls at removing backup files, at the .msp instalation it stalls on time remaining: 15 seconds.
    (a very very long 15 seconds)

  7. Abdelkader

    Hi Kevin,

    Thanks for this post

    After UR2 upgrade, My SCOM Reporting console is blank. This happens only from the UR2 SCOM Console (10.22.10610.0), However, When I run the old one from from desktop (10.22.10337.0), the reports are loaded correctly. Any idea ?

    Regards

    • OdgeUK

      I think I had this issue after 2019 UR5 upgrade. It could only be fixed by reinstalling the Console feature from the SCOM Source media.

  8. Stephen

    I ran through all these steps and still ended up with the linux matching agent kit error. I had gone from RTM to UR2. Seems that some of these hotfixes still need to be applied. One of the hotfixes says it requires UR1 to be installed.

    5028684 requires 5024286 and that requires UR1. Is there a documented path to get from RTM to UR2? Not sure if I install these older KBs over UR2 if I will end up with pre then UR2 files

  9. Piet v.d. Hout

    Our upgrade went well, however if you perform an check for update, both manager as console will be constantly back to be upgraded. It appears there is a mismatch between version installed and version to be checked when perform an update check. We checked this against our WSUS server as well as updates via Microsoft. Are we the only ones having this problem ?

  10. Stev

    I can’t install this update. It always fails while configuring the OperationsManagerDW database:

    UpdateSQLScripts|Executing the task : DatawarehouseUpdateTask
    Exception in UpdateDatabase : System.TimeoutException: The operation has timed out.
    at Microsoft.EnterpriseManagement.Runtime.TaskRuntimeManagement.ExecuteTaskInternal(IEnumerable`1 targets, Guid taskId, TaskConfiguration configuration)
    at Microsoft.EnterpriseManagement.Runtime.TaskRuntimeManagement.ExecuteTask(IEnumerable`1 targets, ManagementPackTask task, TaskConfiguration configuration)
    at Microsoft.MOMv3.Setup.MOMv3ManagedCAs.ExecuteUpdateTask(Session session, ManagementGroup mg, String patchVersion, String serverInstance, String databaseName, String taskName, String dbPath, MonitoringObject targetInstance)
    at Microsoft.MOMv3.Setup.MOMv3ManagedCAs.UpdateDatabase(Session session, String patchVersion, String serverInstance, String databaseName, ManagementGroup mg, String databasePath, String taskName, String sqlFolder, FileLogger sqlFileLogger, MonitoringObject targetInstance)
    UpdateSQLScripts|DW updation failed|Datawarehouse updated Failed

    • Peter

      I’m having the same issue on a system that I upgraded from SCOM 2019. And unfortunately MS support are not being supportive with fixing my services hub access.
      I hope someone here knows the fix?

  11. Tim

    Hey Kevin,
    we have the following Issue after installaing the UR2 for 2022:

    Our TeamsNotification is not working anymore. We have had used the antive Teams Notification from SCOM 2022.

    Microsoft.EnterpriseManagement.HealthService.ModuleException: Could not load file or assembly ‘Azure.Core, Version=1.20.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040) —> System.IO.FileLoadException: Could not load file or assembly ‘Azure.Core, Version=1.20.0.0, Culture=neutral, PublicKeyToken=92742159e12e44c8’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

    Is there a way/ Fix to get the Notification via Teams getting working again ? 🙂

  12. Steven

    Hi Kevin, in mentioned the following:

    The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases. You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.

    The question is will the SQL script updates run and work if you deploy the server updates via WSUS unattended?

  13. Mike

    Hi Kevin,

    after updating to UR2 i cannot open the Report Tasks.

    I got this error:

    Cannot initialize report.
    Value cannot be null.
    Parameter name: GrouList

    Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.

    System.ArgumentNullException: Value cannot be null.
    Parameter name: GroupList
    at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.Parameters.Controls.Monitoring.ReportMonitoringObjectXmlEditorBase.LoadValues(ReportParameterInfoCollection reportParameters)
    at Microsoft.EnterpriseManagement.Mom.Internal.UI.Reporting.Parameters.ReportParameterBlock.LoadValues()
    at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ReportForm.SetReportJob(Object sender, ConsoleJobEventArgs args)

Leave a Reply

Your email address will not be published.