Menu Close

Upgrade from SCOM 2019 to SCOM 2022 Checklist

image

 

This is a planning checklist that will help you determine if an in-place upgrade is possible, and how to prepare the environment in advance for it.  It is similar to my previous post on Upgrading SCOM 2016 to SCOM 2019.

 

1. Verify we are moving from a supported version of SCOM to SCOM 2022.

2. Verify the SQL server versions and service pack levels are supported for both SCOM 2019 and SCOM 2022

3. Verify all OS versions for SCOM server roles will be supported for both SCOM 2019 and SCOM 2022

4. Verify all SERVER ROLES meet minimum hardware sizing for SCOM 2022

5. Verify all AGENT managed Operating Systems are supported for SCOM 2022.

6. Verify all MANAGEMENT PACKS in use are supported for SCOM 2022.

  • Check with 3rd party MP vendors and ensure their MP does not have any known support issues with SCOM 2022. Update these MP’s in advance if required.
  • All previous Microsoft MPs should work with SCOM 2022, although they might not be explicitly documented as supported in SCOM 2022 due to their age.

7. SCOM Database: Verify the OperationsManager database has more than 50 percent free space

8. Optimize Registry settings for management servers

9. Export and review the SCOM management server event logs on all management server roles

  • Look for critical and warning events that indicate major issues that should be resolved before upgrading.
  • Save these for comparison after the upgrade to verify any new issues are actually new

10. Verify SCOM is healthy

  • Review the “Operations Manager > Management Group Health” dashboard in addition to the event logs and ensure SCOM is healthy

11. T-SQL: Clean up the database ETL table in the OperationsManager database

12. Check ALL Management Packs for duplicate Aliases.

  • In previous versions of SCOM, duplicate aliases (with different case) was allowed.  This is no longer allowed.  However, we do not check for this condition as part of the upgrade.
  • Please run the PowerShell or SQL scripts at the bottom of this page in the KNOWN ISSUES section of this article to check for this condition and resolve this BEFORE continuing.

13. SCOM Console: Remove agents from Pending Management

14. Backup unsealed management packs

  • Get a fresh backup of all your unsealed MP’s which contain all your customizations, for disaster recovery
  • Example:    Get-SCOMManagementPack | where {$_.Sealed -eq $false}|Export-SCOMManagementPack -Path c:\mpbackup

15. REMOVE the OMS/Advisor management packs

16. SCOM Console: Disable Notification subscriptions

17. Disable product connectors or any external connections to the SDK.

18. Optional but recommended:  Restart the SQL server service on the OpsDB server and DW server

  • This will kill any stuck or old blocking processes, and free up any buffer cache
  • Wait at least 5 minutes after restarting to ensure the DB’s are online and functioning.
  • Ensure there is no active blocking in the OpsDB before continuing.
  • Consider a reboot of the entire server.

19. Optional but recommended:  Uninstall the SCOM Console and Web Console on the FIRST management server you plan to upgrade and REBOOT.

  • Removing these roles reduce the risk of an upgrade failure.
  • These roles are easy to reinstall once the management group upgrade is completed.

20. Stop the Operations Manager services on ALL Management servers

  • Stop the following services on all management servers in the management group, to ensure NO changes are being made to SQL during the backup, so we can get a good backup right before the upgrade:
  • Microsoft Monitoring Agent
  • System Center Data Access
  • System Center Configuration

21. Backup the SCOM databases

22. Backup the Management Servers

  • Take a VM snapshot or a full bare-metal backup that is restorable, with the SCOM services stopped, so there should be no transient data. This will be for use in the case of disaster recovery only.

23. Remove any old SDK reference software from the management server

  • Some programs install DLL’s that might block upgrade, consider removing them if installed on your management servers:
  • SCOM 2007 R2 Authoring Console
  • Silect MP Author/MP Studio

24. Optional but recommended:  REBOOT ALL Management servers.

  • Rebooting these servers ensures that any OS related issues are observed or cleared before attempting an upgrade.
  • Rebooting these servers helps remove any question that something was wrong with them prior to the upgrade.
  • If a Management server cannot successfully reboot and start up without errors before an upgrade, it certainly cannot after an upgrade.

25. Upgrade the first management server

26. Upgrade additional management servers

  • It is CRITICAL not to upgrade multiple management servers at the same time. You should wait for one to complete FULLY and inspect the logs to ensure it is working, before continuing with the next.

27. Upgrade ACS (if applicable)

28. Upgrade all gateways (if applicable)

29. Upgrade Stand Alone Web Console servers (if applicable)

30. Upgrade Reporting Server

31. Upgrade Stand-Alone Consoles

32. Post Upgrade tasks

33. Reject Pending Management updates for any agents

  • We will update agents later, after applying the latest Update Rollup for SCOM 2022

34. Verify your SCOM license is reporting correctly as licensed

35. Apply the latest Cumulative Update Rollup for SCOM 2022

36. Upgrade Agents

  • Using whatever method you choose, consider upgrading your agents to SCOM 2022 with the latest UR at this point.

 

What to do when things go wrong?

When SCOM upgrades fail, there will be a log telling us why.  Often times you will get an “Error 1603” which is simply a generic error and does not tell you anything.  These log files are typically located in the user profile directory of the account attempting the installation.  C:\Users\<username>\AppData\Local\SCOM\LOGS.  Review ALL the logs, and if needed provide all these logs to a Microsoft engineer when opening a support case.  Log files are not always easy to interpret – but the root cause is almost always in them.

Common issues causing failures:

  • Lack of permissions for the user account performing the upgrade (requires Local admin, SCOM admin, and SQL SysAdmin)
  • TLS 1.2 enforced on management servers or SQL but missing prerequisites
  • A SCOM Agent is installed on a SCOM Management server
  • SQL Database is experiencing blocking from another process.
  • SQL Database does not have enough free space or transaction log space.
  • Advisor MP caused the upgrade to fail because it was not deleted before attempting upgrade

 

Resources:

SCOM 2022 Quickstart Guide

SCOM 2022 Security Accounts Matrix

How to upgrade and update SCOM agents using Tasks

SCOM Gateway Upgrades and Update Rollups made easy

 

Known Issues:

1.  Duplicate Aliases in Management Packs Issue:

  • You MUST check for duplicate aliases. If you have any MP’s with duplicate aliases the SDK will throw errors after the upgrade
  • In previous versions of SCOM, duplicate aliases (with different case) were allowed.  This is no longer allowed.  However, we do not check for this condition as part of the upgrade.
  • If impacted, your SCOM console might be stuck while “connecting”.
  • SDK PowerShell commands might return an error with “An object of class ManagementPackClass with ID <GUID> was not found”
  • Event 27000 In the OpsManager event log : “An Item with the same key has already been added”
  • Failed setup/upgrade installation with messages in the log: ”An Item with the same key has already been added”

Example of a bad MP:

image

To detect MPs with this issue, you can use the following PowerShell or SQL query.  After detection, you will need to export these MP’s and re-label one of the duplicate Aliases, and re-label it anywhere it was used in the body of the XML.

PowerShell:

############################################
#Identify MPs imported with duplicate Aliases
$mps = Get-SCOMManagementPack
foreach ($mp in $mps)
{
  $hashTable = @{}
  foreach ($ref in $mp.References)
  {
    try {$hashTable.Add($ref.Key, $ref.Value)}
    catch
    {
      $MPName = $mp.Name
      $MPDisplayName = $mp.DisplayName
      $MPVersion = $mp.Version
      "MP contains duplicate aliases: Name=($MPName) DiplayName=($MPDisplayName) Version=($MPVersion)"
    }
  }
}
############################################

SQL:

-- LIST ALL MPs that have a duplicate Alias reference
declare
       @mpFriendlyName nvarchar(255),
       @mpName nvarchar(255),
       @mpId uniqueidentifier,
       @mpXml as xml

create table #badMPTable (
       mpId uniqueidentifier,
       mpName nvarchar(255),
       mpFriendlyName nvarchar(255)
)

declare mp_cursor cursor local forward_only read_only for
       select 
              MPFriendlyName,
              MPName,
              ManagementPackId,
              convert(xml, MPXML)
       from ManagementPack

open mp_cursor
fetch next from mp_cursor
into @mpFriendlyName, @mpName, @mpId, @mpXml

while @@FETCH_STATUS = 0 begin
       select
       n.value('@Alias','nvarchar(255)') as mpRef
       into #tempRefs
       from @mpXml.nodes('/ManagementPack/Manifest/References/Reference') as a(n)

       if exists(
              select count(*)
              from #tempRefs
              group by mpRef
              having count(*) > 1
       ) begin
              insert into #badMPTable (
                     mpId,
                     mpName,
                     mpFriendlyName
              ) values (
                     @mpId,
                     @mpName,
                     @mpFriendlyName
              )
       end

       drop table #tempRefs

       fetch next from mp_cursor
       into @mpFriendlyName, @mpName, @mpId, @mpXml
end

close mp_cursor  
deallocate mp_cursor

select *
from #badMPTable

drop table #badMPTable
--End

 

2.  AD integration is broken out of the box in SCOM 2022. 

You will see events such as:

Log Name:      Operations Manager
Source:        Health Service Modules
Event ID:      11460
Description:  There was an error while updating Management Group container.
Exception: System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException
Message: The specified domain does not exist or cannot be contacted.

This is caused by the fact that NT Authority\System is now exposed in the Operations Manager Administrators user role.  It was always present in previous versions of SCOM, but hidden.  Now that it is exposed, it will break AD integration.

image

Remove this, and AD integration works fine.

 

3.  SCOM 2022 Web console Performance and Diagram views might throw a 500 error

  • Need to change security:
  • On the web console server – find the folder: \Program Files\Microsoft System Center\Operations Manager\WebConsole\MonitoringView\TempImages
  • Change the security on this folder for Authenticated Users – and add “Write” capability.

34 Comments

  1. Brian Wright
    Reply

    Would it help the upgrade if I reindexed all the tables in the database before starting?

    SET ANSI_NULLS ON
    SET ANSI_PADDING ON
    SET ANSI_WARNINGS ON
    SET ARITHABORT ON
    SET CONCAT_NULL_YIELDS_NULL ON
    SET QUOTED_IDENTIFIER ON
    SET NUMERIC_ROUNDABORT OFF
    EXEC SP_MSForEachTable “Print ‘Reindexing ‘+’?’ DBCC DBREINDEX (‘?’)”

    • Kevin Holman Post author
      Reply

      So you have BOTH an unsupported SQL deployment, AND an unsupported SCOM deployment.
      Do you always run all your vendor software in unsupported configurations?
      That’s a bold strategy.
      Best wishes.

    • Kevin Holman Post author
      Reply

      I am not aware of any bugs that I’d consider serious enough to wait for UR1. What issues are you talking about that would make you consider this?

      • RH
        Reply

        I talk about all your action points that have to be followed to ensure a problem free upgrade. Many of these are not mentioned in the official upgrade document from Microsoft (https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?source=recommendations&view=sc-om-2022).

        That makes me conclude that this release has not been tested enough before release to the public. Bugs may be the wrong definition, but definitely critical for a SCOM-environment that should operate 24/7 if it fails because MS has not informed about all the items on your checklist. We’re very happy that we waited for your article before we upgrade our production environment 🙂 That’s why I mentioned waitng for UR1, in hope that UR1 fix some of items in the checklist, and makes upgrade easier.

        • Kevin Holman Post author
          Reply

          “notes from the field” have been around for every product release since MOM 2000. This checklist is identical to SCOM 2019, and almost identical for SCOM 216. It is mostly in the upgrade guide, with a few helpful tips or gotchyas learned from the field or support. If you are inferring something about SCOM 2022 because of that…. it hasn’t really changed in 3 releases and 6 years.

          • RH

            Ok,I’ve known about a few of the points in the checklist, but that it has been almost identical since 2016 is not very reassuring. Why haven’t some of these shortcomings been remedied for 6 years? I still maintain my point that the official MS site for upgrade is seriously lacking, and the product has not been tested/bugfixed sufficiently.

    • Kevin Holman Post author
      Reply

      I have not heard of this one. It is possible a bug has already been raised, but this seems like a very minor issue. The fastest way to get a fix, is simply to open a support case and request this be filed as a bug.

  5. Martin
    Reply

    We have SCOM 2019 installled on Windows server 2016 OS and we have SQL Server 2017 with latest CU.
    Please correct me if I’m wrong, MS suppported upgrade theory will then be according to the following order:

    1. Inplace Upgrade OS from “Windows server 2016” to “Windows Server 2019”
    2. Inplace Upgrade “SCOM from 2019” to “SCOM 2022”
    3. Inplace Upgrade OS to Windows Server 2022 (Optional)
    4. Inplace Upgrade SQL Server 2017 to SQL 2019 (Optional)

      • Martin
        Reply

        Thanks for reply Kevin! Do you think it might be possible with future SCOM 2019 URxx to support Windows Server 2022? I’m mainly thinking of avoiding two inplace upgrades for Server OS.

        • Kevin Holman Post author
          Reply

          The only way to do that would be to just leave your servers on WS2016, and upgrade all the way, or move them all to WS2022 right now and just have an unsupported configuration in the interim for a short time, and you’d have to test it well to make sure it even works. We don’t test those scenarios, but they may work ok.

  6. Paul
    Reply

    We are running SCOM 2019 on Windows Server 2016 with SQL 2019 CU17 and would like to upgrade to SCOM 2022. Is it possible to setup new Windows Server 2022 servers, perform the SCOM 2022 setup and connect them to the existing SCOM management group / databases?

    • Kevin Holman Post author
      Reply

      No. Your choices are side by side migration to a NEW management group, or in-place upgrade.

      For an in-place upgrade, you have to jump through multiple steps. You would deploy new management servers on WS2019, and then add them to the management group, and retire anything running on WS2016. Then you can add new WS2022 management servers and SQL servers.

      Unfortunately, our in-place upgrade restrictions being so tight between versions, we make it very difficult for customers to upgrade easily. This is why so many customers opt for side by side migrations to new management groups. An alternative is to just attempt an upgrade in place, then worry about the unsupported OS versions later down the road, requiring less repetitive steps and upgrades. The downside is that it will not be a supported upgrade path because we didn’t test that scenario – so you need to test it fully in a lab environment before attempting production.

      • Paul
        Reply

        Thank you for your detailed answer Kevin!

        One more question. When you say; “You would deploy new management servers on WS2019, and then add them to the management group”, do you mean In-Place upgrade the OS of all existing SCOM 2019 servers running WS2016 to WS2019?
        And the next step is to In-place upgrade the OS to WS2022 or stay on WS2019 and In-place upgrade SCOM to 2022.

        • Kevin Holman Post author
          Reply

          You can do either/or. I work with some customers who will not allow in place upgrades of the OS. So they just add new VM’s on the new OS, and add them as new management servers, move agents and resource pool memberships, and then retire the old management servers. The other option is to simply in-place upgrade the Management server OS.

          Then – once on WS2019 – you can in-place upgrade SCOM to SCOM 2022. Then after that, you can consider upgrading the OS again to WS2022, or add in new management servers on WS2022 VM’s and retire the old WS2019 management servers.

  7. Brian Hansen
    Reply

    My SCOM 2022 console is very slow to load and search groups. I saw a hotfix for this problem for 2019. Is there one for 2022?

    • Kevin Holman Post author
      Reply

      Yes there is – but not published for some reason. Easy to get – open a support case and request the private hotfix for it. Or, wait a week or two and UR1 for SCOM should be out which includes the fix.

  9. Franky
    Reply

    Hi Kevin,

    Our SCOM is 2019 and with SCOM agent 10.19.10185.0 and most of our servers are on .Net is 4.0.

    Therefore, If we upgrade SCOM server to 2022. Can we still keeping using current SCOM agent instead of using SCOM 2022 agent which has more requirement as you mentioned as below.

    SCOM 2022 agent supports Windows 10, Windows 11, Windows Server 2012,2012R2,2016,2019,2022.
    SCOM 2022 agent requires PowerShell 3.0 or later to be installed on ALL agent managed machines.
    SCOM 2022 agent requires .NET 4.7.2 or later. .NET 4.7.2 was installed by default on Windows Server 2019. Older OS versions must be upgraded.

    Thanks.

  10. Bojan Zivkovic
    Reply

    I have SCOM 2019 running on Windows Server 2016 with remote SQL Server 2016 hosting SCOM DBs including report DBs running on Windows Server 2016 – what would be the best way to upgrade to SCOM 2022? Also, since starting with SQL Server 2017, SSRS is separate product is there any procedure how to upgrade SSRS 2016 to SSRS 2019/2022? Regarding SCOM reports I guess they all are in MPs (sealed/unsealed) so nothing to do during SSRS 2016 upgrade?

  11. Bert Pinoy
    Reply

    Hi Kevin,

    Thanks again for your upgrade procedure! It helped me a lot at several customers.
    The latest upgrade I did went quite well (management servers, consoles, etc), except for the Reporting upgrade. When trying to upgrade the Reporting the SCOM wizard gives a “no such host is known” error:

    [11:24:37]: Error: :Uncaught Exception: Threw Exception.Type: System.Net.Sockets.SocketException, Exception Error Code: 0x80004005, Exception.Message: No such host is known
    [11:24:37]: Error: :StackTrace: at System.Net.Dns.GetAddrInfo(String name)
    at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
    at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
    at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.GetReportingSettings()
    at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.IsReportingValidToUpgrade()
    at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.Rationalize()
    at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.OMReportingSetupComponent.RationalizeComponents()
    at Microsoft.SystemCenter.Essentials.SetupFramework.Program.RationalizeUpgrade()
    at Microsoft.SystemCenter.Essentials.SetupFramework.Program.Main()

    I also see in the application log the following:

    Faulting application name: SetupChainerUI.exe, version: 10.22.10118.0, time stamp: 0x6206e038
    Faulting module name: KERNELBASE.dll, version: 10.0.14393.5582, time stamp: 0x63882301
    Exception code: 0xe0434352
    Fault offset: 0x0000000000026ea8
    Faulting process id: 0x4710
    Faulting application start time: 0x01d951a747376f49
    Faulting application path: C:\Users\Username\AppData\Local\SCOM\Setup\SetupChainerUI.exe
    Faulting module path: C:\Windows\System32\KERNELBASE.dll
    Report Id: 267d1a36-5b77-4d71-956e-8b0ed96d3b87
    Faulting package full name:
    Faulting package-relative application ID:

    Is this something you’ve experienced somewhere by any chance?
    The reporting is now still running as SCOM 2019 UR3 and is still functioning. So for now we are not blocked.
    SQL Reporting version is: 14.0.600.1699 (that’s SQL 2017)

    Best regards,
    Bert

  12. Andri
    Reply

    During the upgrade from SCOM 2019 to 2022 I got this error:

    during the “Management server” upgrade part getting these errors to setup log:
    Error: :LaunchMsiSetup: Failed, return code: 1603
    Error: :Error:Failed to uninstall previous MOM product on this machine, cannot proceed with upgrade
    Error: :Error:Failed to uninstall Operations Manager Agent on this machine. This is fatal error, we cannot proceed with upgrade

    Fix was to delete this registry key from all management servers and then it will be success:
    HKEY_CLASSES_ROOT\Installer\UpgradeCodes\C96403E8AD6025B4F9E1FE9C574E34AE

    Information from: http://www.matej.guru/2018/02/26/upgrade-to-scom-1801-failed-to-uninstall-operations-manager-agent/

    Maybe worth to add to known issues section?

  13. Chris
    Reply

    Hi Kevin,
    I’m looking to upgrade to SCOM 2022 from 2019 with SQL Server 2016. Do I need to upgrade the SQL Db first to be able to do an inplace upgrade?
    Kind regards
    Chris

  14. jonathan.costa@whitecube.com.br
    Reply

    i am getting the following error when running custom reports after upgrading from scom 2019 to 2022

    Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.

    System.NullReferenceException: Object reference not set to an instance of an object.
    at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ReportForm.RunReportJob(Object sender, ConsoleJobEventArgs args)

  15. Denis
    Reply

    Hi Kevin, we recently upgraded from SCOM 2019 to 2022 and we are seeing some NTLM authentication coming from our SCOM SSRS server to the DefaultSDKServiceMachine Management server.
    It seem to happen at regular cycle, once every 12hrs or if we reset the SQL Reporting Services.

    All the other auth are using Kerberos and we are not getting any SPN errors.
    rsreportserver.config contains the option already.

    I saw the release note that SCOM 2022 now support NTLM hardened system but no details or anything more.
    This page is not really helping either
    https://learn.microsoft.com/en-us/system-center/scom/deploy-install-reporting-server?view=sc-om-2022#install-reporting-services-on-ntlm-hardened-enterprises

    We did an upgrade from 2019 to 2022 on the reporting server and never saw any option for authentication.

    Any idea on what could cause this?

  16. Brian Wright
    Reply

    In step 8 you say “Optimize Registry settings for management servers”, can SCOM 2019 UR4 run with these settings without issue?

  17. Brian Wright
    Reply

    I have this post in the System Center Q&A, but thought I’d post here as well.

    Kevin, when you say “TLS 1.2 enforced on management servers or SQL but missing prerequisites” in What to do when things go wrong?, does this mean both the OS and OM registry keys mentioned here – https://learn.microsoft.com/en-us/system-center/scom/plan-security-tls12-config?view=sc-om-2022&viewFallbackFrom=sc-om-2022How%20to%20implement%20Transport%20Layer%20Security%201.2
    need to exist on the MSs and SQL servers before upgrading. I also have seen one comment that suggests they need to be on every agent server as well in order for the agents to update from pending management correctly – is this true?

Leave a Reply

Your email address will not be published.