Upgrade from SCOM 2012R2 to SCOM 2016 Checklist

As many customers prepare to move to SCOM 2019, one thing I have been hearing is how many customers are still on SCOM 2012R2.  There is no direct in-place upgrade path from SCOM 2012R2 to SCOM 2019.  You have to upgrade 2012R2 > 2016 > 2019.  Many years ago I wrote an in-place upgrade checklist for SCOM 2012R2 to SCOM 2016, and realized it was never published, and might help some customers during this transition. 

Here goes!

 

1. Verify we are moving from a supported version of SCOM to SCOM 2016.

• SCOM 2012 R2 URxx is currently deployed.
• UR9 is the minimum supported UR for SCOM 2012R2 to upgrade to SCOM 2016
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-overview?view=sc-om-2016#in-place-upgrade
• https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-2016#in-place-upgrade

2. Verify the SQL server versions and service pack levels are supported for both SCOM 2012 R2 and SCOM 2016

• SQL is xx.x.xxxx (SQL 201x SPx) X Edition
• Is this is supported for both SCOM versions
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2016#sql-server-requirements

3. Verify all OS versions for SCOM server roles will be supported for SCOM 2012 R2 and SCOM 2016

• All server roles are on Windows Server 201x
• Is this is supported for both SCOM versions
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-2016#server-operating-system

4. Verify all server roles meet minimum hardware sizing for SCOM 2016

• Are all SCOM and SQL servers are sized at least to minimum specifications.
• Reference: https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-2016#hardware-requirements
• https://aka.ms/scomsizer

5. Verify all agent managed Operating Systems are supported for SCOM 2016.

• SCOM 2016 does not support WS2003 OS for a SCOM 2016 agent.
• SCOM 2016 requires PowerShell to be installed on ALL agent managed machines. Windows Server 2008 (non-R2) did not ship with PowerShell and this should be verified first as PowerShell is a requirement for base agent monitoring and discoveries.
• Reference:  https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
• https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-2016#microsoft-monitoring-agent-operating-system

6. Verify all management packs in use are supported for SCOM 2016.

• Check with 3rd party MP vendors and ensure their MP does not have any known support issues with SCOM 2016. Update these MP’s in advance if required.

7. SCOM Database: Verify the OperationsManager database has more than 50 percent free space

• Ensure the OpsDB has more than 50% free space
• Ensure the Transaction log is pre-sized to 50% of the DB size
• Ensure the database is set to Simple recovery mode to remove any transactional issues during upgrade.
• Ensure the DataWarehouse DB has sufficient free space so it will not have to grow during upgrade.
• This is a good time to verify the TempDB is optimized, and SQL is otherwise healthy and performing well.
• Reference: https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#verify-the-operations-manager-database-has-more-than-50-percent-free
• https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/

8. Optimize Registry settings for management servers

• There are registry updates that will reduce the chance of an upgrade failure (especially the DW deployment setting):
• Reference:  https://kevinholman.com/2017/03/08/recommended-registry-tweaks-for-scom-2016-management-servers/

9. Export and review the SCOM management server event logs on all management server roles

• Look for critical and warning events that indicate major issues that should be resolved before upgrading.
• Save these for comparison after the upgrade to verify any new issues are actually new

10. Verify SCOM is healthy

• Review the “Management Group Health” dashboard in addition to the event logs and ensure all issues are managed

11. T-SQL: Clean up the database ETL table in the OperationsManager database

• This grooms out unnecessary data and speeds up the upgrade, removing a possible timeout scenario.
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#cleanup-the-database-etl-table

12. SCOM Console: Remove agents from Pending Management

• As a best practice, you should not leave agents in pending management. This will consume resources because there are workflows targeting these class instances.
• If you have your agent approval setting to “Review” you should set it to “Reject” so any new agents do not attempt to re-create pending actions.
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#remove-agents-from-pending-management
• It is a good idea to query the AgentPendingActions table at this point, to ensure you don’t have any stale pending actions that do not show up in the console:
• Reference:  https://kevinholman.com/2008/09/29/agent-pending-actions-can-get-out-of-synch-between-the-console-and-the-database/

13. Backup unsealed management packs

• Get a fresh backup of all your unsealed MP’s which contain all your customizations, for disaster recovery
• Example:    Get-SCOMManagementPack | where {$_.Sealed -eq $false}|Export-SCOMManagementPack -Path c:\mpbackup

14. SCOM Console: Disable Notification subscriptions

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#disable-notification-subscriptions
• http://www.scom2k7.com/subscription-tool/

15. Disable product connectors or any external connections to the SDK.

• We do not want any connections to the SDK during upgrade. This includes Product Connector Services, Microsoft Orchestrator, any scheduled tasks you might be running with custom scripts, or the Microsoft Exchange 2010 Correlation engine.
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#disable-connectors

16. REMOVE the OMS/Advisor management packs

• There is a bug in the Advisor MP (Microsoft.SystemCenter.Advisor.mpb) which can break the SCOM upgrade process.
• Manually remove ALL the Advisor/OMS MP’s. 
• Reference:  How to remove OMS and Advisor management packs – Kevin Holman’s Blog

17. Stop the Operations Manager services on Management servers

• Stop the following services on all management servers in the management group, to ensure NO changes are being made to SQL, so we can get a good backup right before the upgrade:
• Microsoft Monitoring Agent
• System Center Data Access
• System Center Configuration

18. Backup the SCOM databases

• Backup the OperationsManager, OperationsManagerDW, and ReportServer databases in case we need to restore due to a failed upgrade.
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2016#backup-the-operations-manager-databases

19. Backup the Management Servers

• Take a VM snapshot or a full bare-metal backup that is restorable, with the SCOM services stopped, so there should be no transient data. This will be for use in the case of disaster recovery only.

20. Install SCOM prerequisites on management servers with consoles

• Report Viewer 2015 runtime
• Microsoft CLR Types for SQL Server 2014
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-1801#operations-manager-console

21. Ensure .Net 3.5, and .Net 4 (or 4.5) are both installed on all management servers

• There is an issue on upgrading which can fail without .Net 3.5, even though this is not a specific prerequisite, so ensure it is installed as well as .NET 4+
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/plan-system-requirements?view=sc-om-1801#management-servergateway-server

22. Remove any old SDK reference software from the management server

• Some programs install DLL’s that might block upgrade, consider removing them if installed on your management servers:
• SCOM 2007 R2 Authoring Console
• Silect MP Author/MP Studio

23. Optional – uninstall the SCOM Console and Web Console on the FIRST management server you plan to upgrade and REBOOT.

• Removing these roles reduce the risk of an upgrade failure.
• These roles are easy to reinstall once the management group upgrade is completed.
• A REBOOT of all servers before attempting an upgrade is recommended. It is hard to know what patches might have been applied already pending reboot, and you want to ensure that the server can reboot reliably before attempting an upgrade.

24. Optional – Restart the SQL server service on the OpsDB server and DW server

• This will kill any stuck or old blocking processes, and free up any buffer cache
• Wait at least 5 minutes after restarting to ensure the DB’s are online and functioning.
• Ensure there is no active blocking in the OpsDB before continuing.
• Consider a reboot.

25. Enable TLS 1.0 on the SQL servers and the Management Servers.

• If you previously disabled TLS 1.0 on the SQL servers and Management servers, this must be temporarily reverted back to enabled.
• SCOM 2016 does not support installation without TLS 1.0. 
• After updating to the latest Update Rollup on SCOM 2016, you can disable TLS 1.0 and TLS 1.1, enforcing TLS 1.2 only.

26. Upgrade the first management server

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-mgmt-server?view=sc-om-2016

27. Upgrade additional management servers

• It is CRITICAL not to upgrade multiple management servers at the same time. You should wait for one to complete FULLY and inspect the logs to ensure it is working, before continuing with the next.

28. Upgrade ACS (if applicable)

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-acs-collector?view=sc-om-2016

29. Upgrade all gateways (if applicable)

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-gateway-server?view=sc-om-2016

30. Upgrade Stand Alone Web Console servers (if applicable)

• If you have web console servers that are NOT on management servers, upgrade those now
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-web-console?view=sc-om-2016

31. Upgrade Reporting Server

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-reporting-server?view=sc-om-2016

32. Upgrade Stand-Alone Consoles

• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-ops-console?view=sc-om-2016

33. Post Upgrade tasks

• Review event logs and look for new errors or issues on all management servers
• Re-enable notification subscriptions
• Re-enable Product Connectors, Remote SDK connections, Custom tasks, Exchange Correlation Engine where applicable.
• Test all key features of SCOM, review Management Group Health Dashboard.
• Reference:  https://docs.microsoft.com/en-us/system-center/scom/deploy-upgrade-post-tasks?view=sc-om-2016

34. Reject Pending Management updates for any agents

• We will update agents later, after applying the latest Update Rollup.

35. Fix scheduled maintenance mode permissions

• Reference:  https://kevinholman.com/2016/10/22/enabling-scheduled-maintenance-in-scom-2016/

36. Verify your SCOM license is reporting correctly as licensed

• Reference:  https://kevinholman.com/2017/06/29/dont-forget-to-license-your-scom-2016-deployments/

37. Apply the latest Cumulative Update Rollup for SCOM 2016

• You should wait 24 hours after an upgrade from SCOM 2012R2 before applying the latest SCOM 2016 update rollup. There are scripts as part of the upgrade that can take several hours to complete, and it is a best practice to not interrupt these.

38. Upgrade Agents

• Using whatever method you choose, consider upgrading your agents to SCOM 2016 with the latest UR at this point.
• Be cautious of the APM issue in SCOM 2016 agent:
• Reference:  https://kevinholman.com/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/
• The APM issue is resolved in current Update Rollups for SCOM, but this must be applied immediately after deploying a SCOM 2016 agent if you have not removed the APM MP’s as discussed in the article above.  If you do not use APM, I recommend removing those MP’s.

39.  (Optional) Disable TLS 1.0 and 1.1 on the SQL servers and Management Servers if you wish to enforce TLS 1.2 only.

• SCOM 2016 supports TLS 1.2 and no longer requires TLS 1.0/1.1 since SCOM 2016 UR4 and later.
• Implementing TLS 1.2 enforcement with SCOM – Kevin Holman’s Blog