Menu Close

Upgrade from SCOM 2016 to SCOM 2019 Checklist

image

 

This is a planning checklist that will help you determine if an in-place upgrade is possible, and how to prepare the environment in advance for it.  It is similar to my previous post on Upgrading SCOM 2012R2 to SCOM 2016.

 

1. Verify we are moving from a supported version of SCOM to SCOM 2019.

2. Verify the SQL server versions and service pack levels are supported for both SCOM 2016/1801/1807 and SCOM 2019

3. Verify all OS versions for SCOM server roles will be supported for both SCOM 2016/1801/1807 and SCOM 2019

4. Verify all SERVER ROLES meet minimum hardware sizing for SCOM 2019

5. Verify all AGENT managed Operating Systems are supported for SCOM 2019.

6. Verify all MANAGEMENT PACKS in use are supported for SCOM 2019.

  • Check with 3rd party MP vendors and ensure their MP does not have any known support issues with SCOM 2019. Update these MP’s in advance if required.

7. SCOM Database: Verify the OperationsManager database has more than 50 percent free space

8. Optimize Registry settings for management servers

9. Export and review the SCOM management server event logs on all management server roles

  • Look for critical and warning events that indicate major issues that should be resolved before upgrading.
  • Save these for comparison after the upgrade to verify any new issues are actually new

10. Verify SCOM is healthy

  • Review the “Operations Manager > Management Group Health” dashboard in addition to the event logs and ensure SCOM is healthy

11. T-SQL: Clean up the database ETL table in the OperationsManager database

12. SCOM Console: Remove agents from Pending Management

13. Backup unsealed management packs

  • Get a fresh backup of all your unsealed MP’s which contain all your customizations, for disaster recovery
  • Example:    Get-SCOMManagementPack | where {$_.Sealed -eq $false}|Export-SCOMManagementPack -Path c:\mpbackup

14. SCOM Console: Disable Notification subscriptions

15. Disable product connectors or any external connections to the SDK.

16. Optional but recommended:  Restart the SQL server service on the OpsDB server and DW server

  • This will kill any stuck or old blocking processes, and free up any buffer cache
  • Wait at least 5 minutes after restarting to ensure the DB’s are online and functioning.
  • Ensure there is no active blocking in the OpsDB before continuing.
  • Consider a reboot of the entire server.

17. Optional but recommended:  Uninstall the SCOM Console and Web Console on the FIRST management server you plan to upgrade and REBOOT.

  • Removing these roles reduce the risk of an upgrade failure.
  • These roles are easy to reinstall once the management group upgrade is completed.

18. Stop the Operations Manager services on Management servers

  • Stop the following services on all management servers in the management group, to ensure NO changes are being made to SQL during the backup, so we can get a good backup right before the upgrade:
  • Microsoft Monitoring Agent
  • System Center Data Access
  • System Center Configuration

19. Backup the SCOM databases

20. Backup the Management Servers

  • Take a VM snapshot or a full bare-metal backup that is restorable, with the SCOM services stopped, so there should be no transient data. This will be for use in the case of disaster recovery only.

21. Install SCOM 2019 prerequisites on management servers with consoles

22. Ensure .Net 3.5, and .Net 4 (or 4.5) are both installed on ALL management servers

23. Remove any old SDK reference software from the management server

  • Some programs install DLL’s that might block upgrade, consider removing them if installed on your management servers:
  • SCOM 2007 R2 Authoring Console
  • Silect MP Author/MP Studio

24. Optional but recommended:  REBOOT ALL Management servers.

  • Rebooting these servers ensures that any OS related issues are observed or cleared before attempting an upgrade.
  • Rebooting these servers helps remove any question that something was wrong with them prior to the upgrade.
  • If a Management server cannot successfully reboot and start up without errors before an upgrade, it certainly cannot after an upgrade.

25. Upgrade the first management server

26. Upgrade additional management servers

  • It is CRITICAL not to upgrade multiple management servers at the same time. You should wait for one to complete FULLY and inspect the logs to ensure it is working, before continuing with the next.

27. Upgrade ACS (if applicable)

28. Upgrade all gateways (if applicable)

29. Upgrade Stand Alone Web Console servers (if applicable)

30. Upgrade Reporting Server

31. Upgrade Stand-Alone Consoles

32. Post Upgrade tasks

33. Reject Pending Management updates for any agents

  • We will update agents later, after applying the latest Update Rollup for SCOM 2019

34. Verify your SCOM license is reporting correctly as licensed

35. Apply the latest Cumulative Update Rollup for SCOM 2019

  • You should generally wait a few hours after an upgrade to SCOM 2019, before applying the latest SCOM 2019 update rollup. There are warehouse scripts as part of the upgrade that can take several hours to complete, and it is a best practice to not interrupt these.

36. Upgrade Agents

  • Using whatever method you choose, consider upgrading your agents to SCOM 2019 with the latest UR at this point.

 

What to do when things go wrong?

When SCOM upgrades fail, there will be a log telling us why.  Often times you will get an “Error 1603” which is simply a generic error and does not tell you anything.  These log files are typically located in the user profile directory of the account attempting the installation.  C:\Users\<username>\AppData\Local\SCOM\LOGS.  Review ALL the logs, and if needed provide all these logs to a Microsoft engineer when opening a support case.  Log files are not always easy to interpret – but the root cause is always in them.

Common issues causing failures:

  • Lack of permissions for the user account performing the upgrade (requires Local admin, SCOM admin, and SQL SysAdmin)
  • TLS 1.2 enforced on management servers or SQL but missing prerequisites
  • A SCOM Agent is installed on a SCOM Management server
  • SQL Database is experiencing blocking from another process.
  • SQL Database does not have enough free space or transaction log space.

 

Resources:

SCOM 2019 is HERE!

Security changes in SCOM 2019 – Log on as a Service

SCOM 2019 Log On As A Service Management Pack Helper

SCOM 2019 Security Accounts Matrix

SCOM 2019 QuickStart Deployment Guide

Leave a Reply

Your email address will not be published. Required fields are marked *