Menu Close

UR1 for SCOM 2022 – Step by Step

image

KB Article for OpsMgr

List of New Features

Download Update Rollup from the Catalog

Download the NEW Simplified Management Server Update EXE

Download Updated UNIX/Linux Management Packs

Recommended hotfix page

New Features:

  • Support for SQL 2022

Key fixes:

  • See the KB article.  There are too many fixes to list here.

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2022 and never applied an update rollup – you can go straight to the latest one available.

 
Let’s get started

From reading the KB article – the order of operations is:

1. Install the update rollup package on the following server infrastructure:

  • Management Servers
  • Audit Collection Servers
  • Web Console Servers
  • Gateway Servers
  • Operations Console Servers
  • Reporting Server

2. Apply Agent Updates

3. Update Unix/Linux MP’s and Agents

 

Management Servers

image

There is a new process for updating management servers that differs from previous versions of SCOM.  Download the single file management server EXE update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates with a UI to show you success/fail and progress.

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I have multiple management servers  My first two management servers hold 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

Notice the EXE file, and a MSP file exist for the Server update.  The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching.  EITHER of these files will patch the server and attempt to run the SQL scripts and MP imports.  The EXE simply provides a visual progress.  That is the primary difference.  I will ONLY demonstrate and recommend the EXE file for the Management Server role update.

Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.

  • ***Note:  One of the changes in SCOM 2019 Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL.  The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases.  You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.

My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.

I will start with KB5020318-amd64-Server.exe

This is a self-extracting executable, that kicks off a simple update tool.  Accept the license terms, and click “Install

This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.

image

If you have an issue – you can review the setup logs:

  • Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
  • SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
  • MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log

 

Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update:  KB5020318-amd64-WebConsole.msp

Next – install the Console Update (make sure your console is closed):   KB5020318-amd64-Console.msp

You can reboot the server at this time if you were prompted to in order to complete the update.  If you were not prompted to, you do not need to.

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

You can use the same EXE file and MSP files (where applicable) you used for the first management server.  The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if they are not needed. 

 

ACS Update: (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

KB5020318-amd64-ACS.msp

image

 

Updating Gateways:

image

Open an elevated command prompt, and run the update:   KB5020318-amd64-Gateway.msp

The update launches a UI and quickly finishes.

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:   KB5020318-amd64-Reporting.msp

 

Apply Post-UR1 SCOM 2022 Console update

Triangular red sign with exclamation point

WARNING – There is a bug in the SCOM 2022 UR1 Console update that you must fix.  There is a post-UR1 update available which resolves an error where the console crashes with Microsoft.Mom.UI.Wrappers.dll messaging.  This crash happens when you try to modify User Profiles, and when you input credentials to approve pending agents.

System.IO.FileLoadException: Could not load file or assembly ‘Microsoft.Mom.UI.Wrappers, Version=10.22.10337.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

File name: ‘Microsoft.Mom.UI.Wrappers, Version=10.22.10337.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’

Simply download the console update fix from: HERE, and extract it.  On any server with a SCOM 2022 UR1 Console, browse to the \Program Files\Microsoft System Center\Operations Manager\Console directory.  Find the Microsoft.Mom.UI.Wrappers.dll and rename it to “Microsoft.Mom.UI.Wrappers.dll.RTM”.  Then copy the new DLL file in that folder:

image

 

 

Update Agents:

image

image There is a issue in SCOM 2022 RTM that you need to fix first before pushing agent updates.

Browse to your \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory and delete the following two files:

  • KB3117586-amd64-Agent.msp
  • KB9999999-amd64-Agent.msp

This appears to be a bug in SCOM 2022 RTM, as these files should not exist.

image

Once these files are deleted – you can continue.

 

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin.  The Agents MUST have “Remotely Manageable” set to “Yes”.

image

You can approve these – which will result in a success or failure message once complete:

image

Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:

image

And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

 

 

Update UNIX/Linux MPs and Agents:

Linux - Wikipedia

You can get the current Unix/Linux MP updates HERE

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

image

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update.  Yours may vary, depending on what previous versions you are upgrading from:

image

These can take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits.   If they are not present or not updated after import, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.

After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:

image

 

Now you can deploy the Linux agent updates:

image

image

image

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://social.technet.microsoft.com/wiki/contents/articles/7375.scom-configuring-sudo-elevation-for-unix-and-linux-monitoring.aspx

 

Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version. 

DON’T FORGET to also apply the port-UR1 SCOM Console update to resolve console crashes, as instructed above.

 

 

Verifying the update

There are new views in the SCOM console to help with this and make this process MUCH easier.  You do need to wait long enough for the discoveries to run in order for these to update the views.

image

image

image

image

image

image

image

image

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update. 

 

Known Issues:

image

1.  Health Explorer and other web console features that use popups no longer work.

  • Make a backup copy of \Program Files\Microsoft System Center\Operations Manager\WebConsole\Dashboard\main.js
  • Edit main.js (using notepad is fine).
  • Search for the string “sandbox” and on the THIRD hit, edit the string immediately after the third “sandbox” adding:  allow-popups allow-forms
  • See image example below:

image

  • Save the file, then clear the browser cache AND restart IIS service (iisreset). 

 

2.  SCOM 2022 RTM has a bug that left two agent update files in \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\amd64 directory.

You need to delete the following two files:

  • KB3117586-amd64-Agent.msp
  • KB9999999-amd64-Agent.msp

 

3.  SCOM 2022 UR1 Console update has a console crash issue that requires a post-UR1 fix for Microsoft.Mom.UI.Wrappers dll.

 

4.  The new OperationsManager DB free space script requires additional permissions.

  • This script was recently updated from VBS to PowerShell, and some additional instance level permissions are required for the SQL server hosting the OperationsManager database
  • If you are missing these permissions, you will see the following event on one of your management servers:
  • Log Name:      Operations Manager
    Source:        Health Service Script
    Event ID:      100
    Level:         Warning
    Description:
    GetOpsMgrDBPercentFreeSpace.ps1 : Exception calling “Fill” with “1” argument(s): “VIEW SERVER STATE permission
    was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.
    VIEW SERVER STATE permission was denied on object ‘server’, database ‘master’.
    The user does not have permission to perform this action.”
    At line:234 char:5
    +     $adp.Fill($dt) | out-null
    +     ~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
        + FullyQualifiedErrorId : SqlException
  • To resolve this, you must grant the “VIEW ANY DEFINITION and “VIEW SERVER STATE” SQL instance level permission to the Management Server Action account SQL login on the SQL instance hosting the OperationsManager database.  If you use AlwaysOn, make sure you set this permission all replica servers.
  • Open the login properties, select “Securables”, and add “View any definition” and “View Server State” permissions:

image

 

5.  When you run the Management server role update, sometimes the tasks fail to update the SQL databases, or perform the Agent Pending Management task.

This can happen when your permissions are not set correctly for your RunAs accounts.  There are 3 tasks that get deployed from the Management Pack included in the Update Rollup:

MP:  Microsoft.SystemCenter.DBUpdateTask

  • Database update task
  • Datawarehouse update task
  • Agent pending management task

The “Database update task” runs as whatever action account is associated with the RunAs Profile:  “Operational Database Account”.   By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account.  Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

The “Datawarehouse update task” runs as whatever action account is associated with the RunAs Profile: “Data Warehouse Account”.  By default the RunAs account “Data Warehouse Action Account” is associated with this profile, so the task will execute as the credential configured in the “Data Warehouse Action Account” RunAs account, which should be the Data Warehouse Write Action account you specified when you installed SCOM.  This credential has a very high privilege to the OperationsManagerDW database (db_owner) so it can modify anything necessary.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

The “Agent pending management task” runs as whatever action account is associated with the RunAs Profile:  “Operational Database Account”.   By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account.  Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful.  It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.

If any of these fail, it is likely that someone has modified the default permissions for your action accounts to the SQL databases, or someone has incorrectly modified the default RunAs profile associations.  Please review your permissions against:  SCOM 2022 Security Matrix

 

Done!

image

51 Comments

  1. Michiel Rens

    Great article again, Kelvin! And thanks for pointing out some known issues.

    I’ve just upgraded a (non-production) environment from 2019 to 2022 and applied UR1 to it afterwards. All seems to work well, except for one REST API method that I’m using to query some performance counters (https://learn.microsoft.com/en-us/rest/api/operationsmanager/data/retrieve-performance-counters?tabs=HTTP). This used to work well on 2019 but now returns a 404 (not found) error:

    http:///OperationsManager/data/performanceCounters/{entityId}
    StatusCode: 404
    File or directory not found. The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.

    Rolling back UR1 for the Web Console does not seem to make a difference, so I fear it has been introduced with the upgrade from 2019 to 2022.

  2. Peter

    Nice article again Kevin, alway enjoy reading it…..so for the console Post-UR1 SCOM 2022 fix….when is ‘comming soon’ 🙂 ? regards peter

    • Kevin Holman

      I don’t know if we block that or not. You’d have to try it. Even if there are “issues” as long as you can get through setup, as long as you deploy UR1 immediately after it should be OK. But from a “supported” perspective, I do not think we support setup of SCOM 2022 GA on SQL 2022. But it might work. 🙂

      • XeroX

        Sadly it does not work. SCOM 2022 setup blocks SQL 2022 fresh setup. Any chance to include the update during setup? As the “tick” check for update prior setup never worked for me.

    • Abdelkader

      Hi IAN,

      I am planning to do the same setup (SQL 2022 /SCOM 2022 RTM and then upgrade to UR1)
      Have you already tried it and have you found any major issues ?

      Regards

  3. Matt Bravo

    Hey Kevin! Are you seeing an issue with the APM transfer health flapping on a daily basis? I am seeing it starting with my update to UR1, with the alert triggering daily. Maybe I need to do some DB pruning?

    The specified number of hours between data warehouse synchronizations has been exceeded.
    Hours: 11
    Workflow name: Microsoft.SystemCenter.Apm.DataTransferRule
    Instance name: Operations Manager APM Data Transfer Service
    Instance ID:
    Management group:

    • Kevin Holman

      I am not. However, one of the FIRST things I do in customer environments is to delete all the APM and Advisor MP’s.

      • Matt Bravo

        Interesting, and first I’m hearing about that. Is it a best practice for most environments unless those MPs are specifically needed?

        • Kevin Holman

          I probably should be more direct about it. I have some blog posts on how to remove them…. but I never say “I ALWAYS do this”.

          APM is a feature that is really out of date and very few customers use anymore…. and given how many issues we caused with it, removing the MP’s removes noise and risk.
          Advisor is an old out of date feature that in my opinion was always half baked and poorly supported… and sometimes caused major issues with SCOM deployments. It also broke change control as your SCOM deployments would download new MP’s whenever Microsoft felt like it with no control, and these workflows potentially ran on all agents. This was just such a huge no-no that I could never get behind it. So out it goes. Too many MP’s in SCOM get imported by default, in my opinion.

          • Matt Bravo

            Ahh, I was not aware APM was legacy. Welp, I see no reason to keep APM around either, so away it goes in my environment.

            Thanks for the insight, as always!

  4. Anthony Bertronski

    As always, thanks Kevin!

    I do have a question on the agents that go into pending approval. I’ve noticed when the agents are updated through our WSUS instance during monthly patching and after the monitored servers restart, they show the updated UR 1 and correct file version. It’s like I didn’t have to run the pending approval. Is the pending approval something equivalent to allowing the UR1 agent msp file to get run at that moment in time when I choose Approve? This is what I’m guessing since I’ve noticed RPC and dynamic high range ports are needed for the management server to be able to finish the approval.

    Short version – If I’ve installed the UR agent file to the monitored servers and they’ve restarted as part of monthly patching, etc. is there no need for me to run the pending approvals on the agents I see in the console?

    I’ve also verified the correct Update Rollup version with your SCOM Management Group Management MP.

    Thanks

    • Anthony Bertronski

      or maybe I’ve been over-thinking it all this time and when the update rollup gets applied through WSUS or manually then it’s good to go and the items shown in the pending management can just be ignored? I was thinking something else needed to be processed by approving the pending management 🙂

      • Kevin Holman

        Us putting agents into pending is a non-intelligent process. By that, I mean that ALL assigned agents are put into pending when an update is applied to a Management server or Gateway, where Remotely Manageable = True.

        If you apply updates through some other mechanism than agent-push/approve, then ignore pending and just REJECT them.

    • Kevin Holman

      These go missing from the catalog whenever the PG is moving the update into Microsoft Update. This happens every time we do an update and typically on lasts for a couple days

  5. Syed Fauvad

    HI Kevin,

    As always thanks for the wonderful, detailed guide and also pointing out the known issues. However, after the upgrade the web console is missing a lot of features like

    1. the health explorer for any object
    2. global search does not work (search for any object or device)
    3. the alert view on the web console for any object
    4. unable to set Maintenance mode to a specific date.

    Please let us know if you have come across this behavior after the upgrade.

      • Forest

        The same here, the same problem after install UR1.
        After we removed and reinstalled the Web Console role from the RTM installer everything is fine.
        Tried to upgrade again with the UR1 but buggy again, so we went back again to RTM,
        @Charlez: Have you (or MS) found the resolution?

        Thank you,
        Forest

  6. Stephen Morrison

    KB3117586-amd64-Agent.msp
    KB9999999-amd64-Agent.msp

    Whats the impact if these are left. I don’t see this same comment in the 2022 quick start guide or the upgrade from 2019 to 2022 guide. I just stumbled across it here because I was searching the console crash error that you listed.

    • Mike

      Thank you, Stephen. This is very true. Had to delete those two KBs and then my agent version for Gateway server and Management Server started to match. And once again, thank you Kevin! You put up awesome guides and actually explain who things operate very well. God bless your soul!

  7. Christian Jacoby

    Hi Kevin,

    thanks for the brilliant description, as always!

    There’s a small downside with the .exe for the server update. I download the msp from here: https://www.catalog.update.microsoft.com/Search.aspx?q=5020318

    There I have the sha1 in every file name, as it’s standard for the update catalog. But the .exe isn’t in there, so I have to go the long way and download twice in different environments, create the hashes and compare them to make sure I have a clean source…

    I think that’s something that should be way more common nowadays… it’s no big technical challenge and can be done fairly automatically.

  8. Pavel

    Hi Kevin,

    could you please advise me on reinstalling SCOM 2022 UR1 over SQL 2022?

    The SCOM environment contains 2 physical servers (on each of them there is SQL 2022 in AlwaysOn mode and SCOM 2022 UR1 management server). The customer wanted to upgrade the OS (from WS2019 to WS2022) by a clean reinstall without using an in-place upgrade.

    I removed one of SCOM management server (according to the removal procedure, I still see it in the SCOM view of Windows Servers), then SQL, reinstalled the OS and SQL, added it to the cluster with SQL AlwaysOn running successfully, but I cannot install SCOM 2022.

    I ran into the problem is that the SCOM 2022 installation dialog does not offer me an existing database to select after choosing to add SCOM management server. My user has the SQL SYSADMIN role and is also a local server administrator. I assume the problem is that SQL is version 2022 (upgraded to this version after applying UR1 for SCOM 2022), but the SCOM 2022 installer does not include UR1. What should be the correct course of action?

    Here are the errors contained in the “OpsMgrSetupWizard” file:
    Error: Error:Failed to execute sql command. Setup will not retry on this Sql error. Command: p_MOMManagementGroupInfoSelect
    Error: Sql error: 16. Error: 2812. Error Message: Could not find stored procedure ‘p_MOMManagementGroupInfoSelect’.

    From the SQL management studio, I log in to both SQL nodes and the names of the listeners / AG without any problems. The mentioned procedure is present in the DB.

    The entire environment is completely backed up.

    Thank you, Pavel

  9. David S

    KB5020318-amd64-WebConsole.msp – once applied the web console appears blank. Is there a fix for this? Or can the patch be removed?

  10. Christian V

    In regards to the known issue: “The new OperationsManager DB free space script requires additional permissions.”, our DBA team has their own monitoring of the SQL server and is reluctant to add these additional permissions. Is it possible to disable the workflow containing this script?

    • Christian

      I disabled (via override) the monitor “Operational Database Space Free (%)” applying to target “Operations Manager Operational Database Watcher”. Confirmed via systemcenter.wiki that this monitor uses that module via the Operational Database percent free space monitor type. Kevin Holman warned in another forum that SQL monitoring does not replace this monitor and there needs to be at least 40% free space in the operationsmanager database and that the monitor would go critical at 20%, so I’ve informed my SQL DBA team. Thanks, Kevin!

  11. Gerben Plantinga

    Hi Kevin,
    In installed this update and everything went fine accept for the agent update?
    When applying KB5020318-amd64-agent.msp i get an error the the patch cannot be installed because the program to be upgraded may be missing?
    The agent stuck at version 10.20.18067.0

  12. Mike

    Hi, Kevin,

    After I removed those two MSI’s that are not needed, now, when i have a need to deploy the agent manually to only a small subset of servers (there are reasons for that unfortunately) by installing the MOMAgent file, it installs the agent on the older version than UR1. Do you know if i can generate a MOMAgent file for the new version somewhere in the console?

    • Kevin Holman

      That’s not how SCOM agents work. There is no SINGLE MSI file that includes the agent and the Update Rollup – never has been. For manual agent installations, you deploy the agent (SCOM 2022) and then apply the Update Rollup (UR1) as a separate operation.

      When you do a PUSH deployment from the console – it applies both, however, they are called one after the other just like you could script. When you do a push, we deploy the MOMAgentinstaller.exe, the MOMAgent.msi, and the patch file. MomAgentinstaller.exe calls the agent install and the patch file in sequence. I have asked the PG if we could get a command line to do this ourselves for manual agent installations, and they said that MOMAgentInstaller.exe is integrated into the console code, and not meant to be called as a stand-alone installer.

      • Mike

        I understand now. Thank you, Kevin. I ran the UR1 for the agent now on the server with the client that was with the version before UR1 and its now all good. Thank you again for all the help as always. You rock!

  13. John G

    Hi, Kevin!

    We’re having issues with a newly setup SCOM 2022.
    The UR1 has been installed on all Management Servers (plus SQL servers etc).
    When trying to run discovery and install an agent, we get the error regarding “Microsoft.Mom.UI.Wrappers.dll” (System.IO.FileLoadException: Could not load file or assembly ‘Microsoft.Mom.UI.Wrappers, Version=10.22.10118.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The located assembly’s manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
    File name: ‘Microsoft.Mom.UI.Wrappers, Version=10.22.10118.0, Culture=neutral…).
    This is with the hotfix installed. Do you have any ideas on how to solve this?

    Best regards

      • Aidan

        First pass of UR2 from exe on a all-in-one test server was a fail for me. Log said DataAccessService not running (it appeared to be), and an sql syntax error in the sql script log. Management binaries are updated (604), but exe won’t continue on to remaining installs so other components are still 118.. It appears to be running, but I have a snapshot to roll back to if needed.

        • Kevin Holman

          There is a known issue in UR2 with the SQL script. It will likely be re-written. You can manually apply the MSP and then manually apply the SQL script if you don’t wish to wait for the formal update.

  14. sanjivram

    Our organisation is currently using SCOM2012 and we are proposing for SCOM2022, could someone share me a prestation on SCOM2022 please.

  15. JumpingYang

    Share:
    for web console cannot open links: I clear web browser cookies and histories, then it works.
    for web console appear blank, I clear web browser cookies and histories, then reboot machine?

Leave a Reply

Your email address will not be published.