This is a planning checklist that will help you determine if an in-place upgrade is possible, and how to prepare the environment in advance for it. It is similar to my previous post on Upgrading SCOM 2016 to SCOM 2019.
1. Verify we are moving from a supported version of SCOM 2019 to SCOM 2022.
- ONLY SCOM 2019 can be upgraded to SCOM 2022.
- Although it is possible to upgrade from 2019 RTM directly to 2022, it’s STRONGLY recommended to be on Update Rollup 3 or higher before upgrading.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-overview?view=sc-om-2022
2. Verify the SQL server versions and service pack levels are supported for both SCOM 2019 and SCOM 2022
- DOCUMENT: SQL is xx.x.xxxx (SQL 201x SPx) X Edition
- Is this is supported for BOTH SCOM versions?
- SCOM 2019 supports SQL 2016, SQL 2017, and SQL 2019 CU8 or later.
- SCOM 2022 supports SQL 2017 and SQL 2019 CU8 or later, and SQL 2022 CU11 or later.
- Therefore – your SQL servers MUST be either SQL 2017 or SQL 2019 CU8 or later for a supported in place upgrade. If you are running SCOM 2019 on SQL 2016 you must upgrade SQL first.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2022#sql-server-requirements
3. Verify all OS versions for SCOM server roles will be supported for both SCOM 2019 and SCOM 2022
- DOCUMENT: All server roles are on Windows Server 20xx
- Is this is supported for both SCOM versions?
- SCOM 2019 supports Windows Server 2016 and Windows Server 2019 for SCOM server roles.
- SCOM 2022 supports Windows Server 2019 and Windows Server 2022 for SCOM server roles.
- Therefore, your SCOM Server roles MUST be on Windows Server 2019 for a supported in place upgrade.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/system-requirements?view=sc-om-2022#server-operating-system
4. Verify all SERVER ROLES meet minimum hardware sizing for SCOM 2022
- Are all SCOM and SQL servers are sized at least to minimum specifications. Use the Sizing Helper spreadsheet for recommended sizing.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/system-requirements?view=sc-om-2022#hardware-requirements
- https://aka.ms/scomsizer
5. Verify all AGENT managed Operating Systems are supported for SCOM 2022.
- SCOM 2022 agent supports Windows 10, Windows 11, Windows Server 2012,2012R2,2016,2019,2022.
- SCOM 2022 agent requires PowerShell 3.0 or later to be installed on ALL agent managed machines.
- SCOM 2022 agent requires .NET 4.7.2 or later. .NET 4.7.2 was installed by default on Windows Server 2019. Older OS versions must be upgraded.
- You can use the SCOM.Management MP to easily check your agent’s .NET versions: https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
- https://learn.microsoft.com/en-us/system-center/scom/system-requirements?view=sc-om-2022#microsoft-monitoring-agent-operating-system
6. Verify all MANAGEMENT PACKS in use are supported for SCOM 2022.
- REMOVE legacy unsupported UNIX management packs, as these are not supported and will cause issues in SCOM 2022: AIX, Solaris, RHEL5, RHEL6, RHEL7(PPC), Debian 8, SLES11, and SLES12(PPC).
- Check with 3rd party MP vendors and ensure their MP does not have any known support issues with SCOM 2022. Update these MP’s in advance if required.
- All previous Microsoft MPs with the exception of UNIX/Linux MPs should work with SCOM 2022, although they might not be explicitly documented as supported in SCOM 2022 due to their age.
7. SCOM Database: Verify the OperationsManager database has more than 50 percent free space
- Ensure the OpsDB has more than 50% free space
- Ensure the Transaction log is pre-sized to 50% of the DB size
- Ensure the DataWarehouse DB has sufficient free space so it will not have to grow during upgrade.
- This is a good time to verify the TempDB is optimized, and SQL is otherwise healthy and performing well.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#verify-the-operations-manager-database-has-more-than-50-percent-free
- https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
8. Optimize Registry settings for management servers
- There are registry updates that will reduce the chance of an upgrade failure (especially the DW deployment setting):
- Reference: https://kevinholman.com/2017/03/08/recommended-registry-tweaks-for-scom-2016-management-servers/
9. Export and review the SCOM management server event logs on all management server roles
- Look for critical and warning events that indicate major issues that should be resolved before upgrading.
- Save these for comparison after the upgrade to verify any new issues are actually new
10. Verify SCOM is healthy
- Review the “Operations Manager > Management Group Health” dashboard in addition to the event logs and ensure SCOM is healthy
11. T-SQL: Clean up the database ETL table in the OperationsManager database
- Run this SQL script. This grooms out unnecessary data and speeds up the upgrade, removing a possible timeout scenario.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#cleanup-the-database-etl-table
12. Check ALL Management Packs for duplicate Aliases.
- In previous versions of SCOM, duplicate aliases (with different case) was allowed. This is no longer allowed. However, we do not check for this condition as part of the upgrade.
- Please run the PowerShell or SQL scripts at the bottom of this page in the KNOWN ISSUES section of this article to check for this condition and resolve this BEFORE continuing.
13. SCOM Console: Remove agents from Pending Management
- As a best practice, you should not leave agents in pending management. This will consume resources because there are workflows targeting these class instances.
- If you have your agent approval setting to “Review” you should set it to “Reject” so any new agents do not attempt to re-create pending actions.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#remove-agents-from-pending-management
- It is a good idea to query the AgentPendingActions table at this point, to ensure you don’t have any stale pending actions that do not show up in the console:
- Reference: https://kevinholman.com/2008/09/29/agent-pending-actions-can-get-out-of-synch-between-the-console-and-the-database/
14. Backup unsealed management packs
- Get a fresh backup of all your unsealed MP’s which contain all your customizations, for disaster recovery
- Example: Get-SCOMManagementPack | where {$_.Sealed -eq $false}|Export-SCOMManagementPack -Path c:\mpbackup
15. REMOVE the OMS/Advisor management packs
- There is a bug in the Advisor MP (Microsoft.SystemCenter.Advisor.mpb) which can break the SCOM upgrade process.
- Manually remove ALL the Advisor/OMS MP’s.
- Reference: How to remove OMS and Advisor management packs – Kevin Holman’s Blog
16. SCOM Console: Disable Notification subscriptions
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#disable-notification-subscriptions
- http://www.scom2k7.com/subscription-tool/
17. Disable product connectors or any external connections to the SDK.
- We do not want any connections to the SDK during upgrade. This includes Product Connector Services, Microsoft Orchestrator, any scheduled tasks you might be running with custom scripts, or the Microsoft Exchange 2010 Correlation engine.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#disable-connectors
18. Optional but recommended: Restart the SQL server service on the OpsDB server and DW server
- This will kill any stuck or old blocking processes, and free up any buffer cache
- Wait at least 5 minutes after restarting to ensure the DB’s are online and functioning.
- Ensure there is no active blocking in the OpsDB before continuing.
- Consider a reboot of the entire server.
19. Optional but recommended: Uninstall the SCOM Console and Web Console on the FIRST management server you plan to upgrade and REBOOT.
- Removing these roles reduce the risk of an upgrade failure.
- These roles are easy to reinstall once the management group upgrade is completed.
20. Stop the Operations Manager services on ALL Management servers
- Stop the following services on all management servers in the management group, to ensure NO changes are being made to SQL during the backup, so we can get a good backup right before the upgrade:
- Microsoft Monitoring Agent
- System Center Data Access
- System Center Configuration
21. Backup the SCOM databases
- Backup the OperationsManager, OperationsManagerDW, and ReportServer databases in case we need to restore due to a failed upgrade.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?view=sc-om-2022#back-up-the-operations-manager-databases
22. Backup the Management Servers
- Take a VM snapshot or a full bare-metal backup that is restorable, with the SCOM services stopped, so there should be no transient data. This will be for use in the case of disaster recovery only.
23. Remove any old SDK reference software from the management server
- Some programs install DLL’s that might block upgrade, consider removing them if installed on your management servers:
- SCOM 2007 R2 Authoring Console
- Silect MP Author/MP Studio
24. Optional but recommended: REBOOT ALL Management servers.
- Rebooting these servers ensures that any OS related issues are observed or cleared before attempting an upgrade.
- Rebooting these servers helps remove any question that something was wrong with them prior to the upgrade.
- If a Management server cannot successfully reboot and start up without errors before an upgrade, it certainly cannot after an upgrade.
25. Upgrade the first management server
26. Upgrade additional management servers
- It is CRITICAL not to upgrade multiple management servers at the same time. You should wait for one to complete FULLY and inspect the logs to ensure it is working, before continuing with the next.
27. Upgrade ACS (if applicable)
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-acs-collector?view=sc-om-2022
28. Upgrade all gateways (if applicable)
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-gateway-server?view=sc-om-2022
29. Upgrade Stand Alone Web Console servers (if applicable)
- If you have web console servers that are NOT on management servers, upgrade those now
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-web-console?view=sc-om-2016&viewFallbackFrom=sc-om-2022
30. Upgrade Reporting Server
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-reporting-server?view=sc-om-2022
31. Upgrade Stand-Alone Consoles
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-ops-console?view=sc-om-2022
32. Post Upgrade tasks
- Review event logs and look for new errors or issues on all management servers
- Re-enable notification subscriptions
- Re-enable Product Connectors, Remote SDK connections, Custom tasks, Exchange Correlation Engine where applicable.
- Test all key features of SCOM, review Management Group Health Dashboard.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-post-tasks?view=sc-om-2022
33. Reject Pending Management updates for any agents
- We will update agents later, after applying the latest Update Rollup for SCOM 2022
34. Verify your SCOM license is reporting correctly as licensed
35. Apply the latest Cumulative Update Rollup for SCOM 2022
- You should generally wait a few hours after an upgrade to SCOM 2022, before applying the latest SCOM 2022 update rollup. There are warehouse scripts as part of the upgrade that can take several hours to complete, and it is a best practice to not interrupt these.
- Reference: https://learn.microsoft.com/en-us/system-center/scom/release-build-versions?view=sc-om-2022
36. Upgrade Agents
- Using whatever method you choose, consider upgrading your agents to SCOM 2022 with the latest UR at this point.
What to do when things go wrong?
When SCOM upgrades fail, there will be a log telling us why. Often times you will get an “Error 1603” which is simply a generic error and does not tell you anything. These log files are typically located in the user profile directory of the account attempting the installation. C:\Users\<username>\AppData\Local\SCOM\LOGS. Review ALL the logs, and if needed provide all these logs to a Microsoft engineer when opening a support case. Log files are not always easy to interpret – but the root cause is almost always in them.
Common issues causing failures:
- Lack of permissions for the user account performing the upgrade (requires Local admin, SCOM admin, and SQL SysAdmin)
- TLS 1.2 enforced on management servers or SQL but missing prerequisites
- A SCOM Agent is installed on a SCOM Management server
- SQL Database is experiencing blocking from another process.
- SQL Database does not have enough free space or transaction log space.
- Advisor MP caused the upgrade to fail because it was not deleted before attempting upgrade
Resources:
SCOM 2022 Security Accounts Matrix
How to upgrade and update SCOM agents using Tasks
SCOM Gateway Upgrades and Update Rollups made easy
Known Issues:
1. Duplicate Aliases in Management Packs Issue:
- You MUST check for duplicate aliases. If you have any MP’s with duplicate aliases the SDK will throw errors after the upgrade
- In previous versions of SCOM, duplicate aliases (with different case) were allowed. This is no longer allowed. However, we do not check for this condition as part of the upgrade.
- If impacted, your SCOM console might be stuck while “connecting”.
- SDK PowerShell commands might return an error with “An object of class ManagementPackClass with ID <GUID> was not found”
- Event 27000 In the OpsManager event log : “An Item with the same key has already been added”
- Failed setup/upgrade installation with messages in the log: ”An Item with the same key has already been added”
Example of a bad MP:
To detect MPs with this issue, you can use the following PowerShell or SQL query. After detection, you will need to export these MP’s and re-label one of the duplicate Aliases, and re-label it anywhere it was used in the body of the XML.
PowerShell:
############################################ #Identify MPs imported with duplicate Aliases $mps = Get-SCOMManagementPack foreach ($mp in $mps) { $hashTable = @{} foreach ($ref in $mp.References) { try {$hashTable.Add($ref.Key, $ref.Value)} catch { $MPName = $mp.Name $MPDisplayName = $mp.DisplayName $MPVersion = $mp.Version "MP contains duplicate aliases: Name=($MPName) DiplayName=($MPDisplayName) Version=($MPVersion)" } } } ############################################
SQL:
-- LIST ALL MPs that have a duplicate Alias reference declare @mpFriendlyName nvarchar(255), @mpName nvarchar(255), @mpId uniqueidentifier, @mpXml as xml create table #badMPTable ( mpId uniqueidentifier, mpName nvarchar(255), mpFriendlyName nvarchar(255) ) declare mp_cursor cursor local forward_only read_only for select MPFriendlyName, MPName, ManagementPackId, convert(xml, MPXML) from ManagementPack open mp_cursor fetch next from mp_cursor into @mpFriendlyName, @mpName, @mpId, @mpXml while @@FETCH_STATUS = 0 begin select n.value('@Alias','nvarchar(255)') as mpRef into #tempRefs from @mpXml.nodes('/ManagementPack/Manifest/References/Reference') as a(n) if exists( select count(*) from #tempRefs group by mpRef having count(*) > 1 ) begin insert into #badMPTable ( mpId, mpName, mpFriendlyName ) values ( @mpId, @mpName, @mpFriendlyName ) end drop table #tempRefs fetch next from mp_cursor into @mpFriendlyName, @mpName, @mpId, @mpXml end close mp_cursor deallocate mp_cursor select * from #badMPTable drop table #badMPTable --End
2. AD integration is broken out of the box in SCOM 2022.
You will see events such as:
Log Name: Operations Manager
Source: Health Service Modules
Event ID: 11460
Description: There was an error while updating Management Group container.
Exception: System.DirectoryServices.ActiveDirectory.ActiveDirectoryObjectNotFoundException
Message: The specified domain does not exist or cannot be contacted.
This is caused by the fact that NT Authority\System is now exposed in the Operations Manager Administrators user role. It was always present in previous versions of SCOM, but hidden. Now that it is exposed, it will break AD integration.
Remove this, and AD integration works fine.
3. SCOM 2022 Web console Performance and Diagram views might throw a 500 error
- Need to change security:
- On the web console server – find the folder: \Program Files\Microsoft System Center\Operations Manager\WebConsole\MonitoringView\TempImages
- Change the security on this folder for Authenticated Users – and add “Write” capability.
Would it help the upgrade if I reindexed all the tables in the database before starting?
SET ANSI_NULLS ON
SET ANSI_PADDING ON
SET ANSI_WARNINGS ON
SET ARITHABORT ON
SET CONCAT_NULL_YIELDS_NULL ON
SET QUOTED_IDENTIFIER ON
SET NUMERIC_ROUNDABORT OFF
EXEC SP_MSForEachTable “Print ‘Reindexing ‘+’?’ DBCC DBREINDEX (‘?’)”
1. It certainly won’t hurt, but isn’t necessary.
2. You SHOULD be performing a full DB reindex of the OperationsManager DB every night already – so this really should not be a requirement.
https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
I have upgraded to SCOM 2022 from 2019 with SQL server 2016 SP1. So far I don’t see any problems.
So you have BOTH an unsupported SQL deployment, AND an unsupported SCOM deployment.
Do you always run all your vendor software in unsupported configurations?
That’s a bold strategy.
Best wishes.
What a bug-ridden release SCOM 2022 seems to be. Is it better to wait for UR1?
I am not aware of any bugs that I’d consider serious enough to wait for UR1. What issues are you talking about that would make you consider this?
I talk about all your action points that have to be followed to ensure a problem free upgrade. Many of these are not mentioned in the official upgrade document from Microsoft (https://learn.microsoft.com/en-us/system-center/scom/deploy-upgrade-pretasks?source=recommendations&view=sc-om-2022).
That makes me conclude that this release has not been tested enough before release to the public. Bugs may be the wrong definition, but definitely critical for a SCOM-environment that should operate 24/7 if it fails because MS has not informed about all the items on your checklist. We’re very happy that we waited for your article before we upgrade our production environment 🙂 That’s why I mentioned waitng for UR1, in hope that UR1 fix some of items in the checklist, and makes upgrade easier.
“notes from the field” have been around for every product release since MOM 2000. This checklist is identical to SCOM 2019, and almost identical for SCOM 216. It is mostly in the upgrade guide, with a few helpful tips or gotchyas learned from the field or support. If you are inferring something about SCOM 2022 because of that…. it hasn’t really changed in 3 releases and 6 years.
Ok,I’ve known about a few of the points in the checklist, but that it has been almost identical since 2016 is not very reassuring. Why haven’t some of these shortcomings been remedied for 6 years? I still maintain my point that the official MS site for upgrade is seriously lacking, and the product has not been tested/bugfixed sufficiently.
Hi Kevin, do you have any new information about bug described on SCOM Community pages?https://feedback.azure.com/d365community/idea/649aebc2-55e2-ec11-a81b-6045bd796569
Any information about an expected solution or workaround?
Best regards.
I have not heard of this one. It is possible a bug has already been raised, but this seems like a very minor issue. The fastest way to get a fix, is simply to open a support case and request this be filed as a bug.
We have SCOM 2019 installled on Windows server 2016 OS and we have SQL Server 2017 with latest CU.
Please correct me if I’m wrong, MS suppported upgrade theory will then be according to the following order:
1. Inplace Upgrade OS from “Windows server 2016” to “Windows Server 2019”
2. Inplace Upgrade “SCOM from 2019” to “SCOM 2022”
3. Inplace Upgrade OS to Windows Server 2022 (Optional)
4. Inplace Upgrade SQL Server 2017 to SQL 2019 (Optional)
Your assumptions/plan are spot on.
Thanks for reply Kevin! Do you think it might be possible with future SCOM 2019 URxx to support Windows Server 2022? I’m mainly thinking of avoiding two inplace upgrades for Server OS.
The only way to do that would be to just leave your servers on WS2016, and upgrade all the way, or move them all to WS2022 right now and just have an unsupported configuration in the interim for a short time, and you’d have to test it well to make sure it even works. We don’t test those scenarios, but they may work ok.
We are running SCOM 2019 on Windows Server 2016 with SQL 2019 CU17 and would like to upgrade to SCOM 2022. Is it possible to setup new Windows Server 2022 servers, perform the SCOM 2022 setup and connect them to the existing SCOM management group / databases?
No. Your choices are side by side migration to a NEW management group, or in-place upgrade.
For an in-place upgrade, you have to jump through multiple steps. You would deploy new management servers on WS2019, and then add them to the management group, and retire anything running on WS2016. Then you can add new WS2022 management servers and SQL servers.
Unfortunately, our in-place upgrade restrictions being so tight between versions, we make it very difficult for customers to upgrade easily. This is why so many customers opt for side by side migrations to new management groups. An alternative is to just attempt an upgrade in place, then worry about the unsupported OS versions later down the road, requiring less repetitive steps and upgrades. The downside is that it will not be a supported upgrade path because we didn’t test that scenario – so you need to test it fully in a lab environment before attempting production.
Thank you for your detailed answer Kevin!
One more question. When you say; “You would deploy new management servers on WS2019, and then add them to the management group”, do you mean In-Place upgrade the OS of all existing SCOM 2019 servers running WS2016 to WS2019?
And the next step is to In-place upgrade the OS to WS2022 or stay on WS2019 and In-place upgrade SCOM to 2022.
You can do either/or. I work with some customers who will not allow in place upgrades of the OS. So they just add new VM’s on the new OS, and add them as new management servers, move agents and resource pool memberships, and then retire the old management servers. The other option is to simply in-place upgrade the Management server OS.
Then – once on WS2019 – you can in-place upgrade SCOM to SCOM 2022. Then after that, you can consider upgrading the OS again to WS2022, or add in new management servers on WS2022 VM’s and retire the old WS2019 management servers.
Clear, thank you very much Kevin.
My SCOM 2022 console is very slow to load and search groups. I saw a hotfix for this problem for 2019. Is there one for 2022?
Yes there is – but not published for some reason. Easy to get – open a support case and request the private hotfix for it. Or, wait a week or two and UR1 for SCOM should be out which includes the fix.
Step 33 – as there is no UR for 2022 at this point then approving all pending managements is fine
My Bad – just noticed UR1
Hi Kevin,
Our SCOM is 2019 and with SCOM agent 10.19.10185.0 and most of our servers are on .Net is 4.0.
Therefore, If we upgrade SCOM server to 2022. Can we still keeping using current SCOM agent instead of using SCOM 2022 agent which has more requirement as you mentioned as below.
SCOM 2022 agent supports Windows 10, Windows 11, Windows Server 2012,2012R2,2016,2019,2022.
SCOM 2022 agent requires PowerShell 3.0 or later to be installed on ALL agent managed machines.
SCOM 2022 agent requires .NET 4.7.2 or later. .NET 4.7.2 was installed by default on Windows Server 2019. Older OS versions must be upgraded.
Thanks.
I have SCOM 2019 running on Windows Server 2016 with remote SQL Server 2016 hosting SCOM DBs including report DBs running on Windows Server 2016 – what would be the best way to upgrade to SCOM 2022? Also, since starting with SQL Server 2017, SSRS is separate product is there any procedure how to upgrade SSRS 2016 to SSRS 2019/2022? Regarding SCOM reports I guess they all are in MPs (sealed/unsealed) so nothing to do during SSRS 2016 upgrade?
Hi Kevin,
Thanks again for your upgrade procedure! It helped me a lot at several customers.
The latest upgrade I did went quite well (management servers, consoles, etc), except for the Reporting upgrade. When trying to upgrade the Reporting the SCOM wizard gives a “no such host is known” error:
[11:24:37]: Error: :Uncaught Exception: Threw Exception.Type: System.Net.Sockets.SocketException, Exception Error Code: 0x80004005, Exception.Message: No such host is known
[11:24:37]: Error: :StackTrace: at System.Net.Dns.GetAddrInfo(String name)
at System.Net.Dns.InternalGetHostByName(String hostName, Boolean includeIPv6)
at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.GetReportingSettings()
at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.IsReportingValidToUpgrade()
at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.RationalizeReportingUpgrade.Rationalize()
at Microsoft.EnterpriseManagement.OperationsManager.Setup.Common.OMReportingSetupComponent.RationalizeComponents()
at Microsoft.SystemCenter.Essentials.SetupFramework.Program.RationalizeUpgrade()
at Microsoft.SystemCenter.Essentials.SetupFramework.Program.Main()
I also see in the application log the following:
Faulting application name: SetupChainerUI.exe, version: 10.22.10118.0, time stamp: 0x6206e038
Faulting module name: KERNELBASE.dll, version: 10.0.14393.5582, time stamp: 0x63882301
Exception code: 0xe0434352
Fault offset: 0x0000000000026ea8
Faulting process id: 0x4710
Faulting application start time: 0x01d951a747376f49
Faulting application path: C:\Users\Username\AppData\Local\SCOM\Setup\SetupChainerUI.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 267d1a36-5b77-4d71-956e-8b0ed96d3b87
Faulting package full name:
Faulting package-relative application ID:
Is this something you’ve experienced somewhere by any chance?
The reporting is now still running as SCOM 2019 UR3 and is still functioning. So for now we are not blocked.
SQL Reporting version is: 14.0.600.1699 (that’s SQL 2017)
Best regards,
Bert
During the upgrade from SCOM 2019 to 2022 I got this error:
during the “Management server” upgrade part getting these errors to setup log:
Error: :LaunchMsiSetup: Failed, return code: 1603
Error: :Error:Failed to uninstall previous MOM product on this machine, cannot proceed with upgrade
Error: :Error:Failed to uninstall Operations Manager Agent on this machine. This is fatal error, we cannot proceed with upgrade
Fix was to delete this registry key from all management servers and then it will be success:
HKEY_CLASSES_ROOT\Installer\UpgradeCodes\C96403E8AD6025B4F9E1FE9C574E34AE
Information from: http://www.matej.guru/2018/02/26/upgrade-to-scom-1801-failed-to-uninstall-operations-manager-agent/
Maybe worth to add to known issues section?
I experienced the same issue.
I experienced the same issue and the reg key deletion fixed the issue. Not sure what that key’s significance is. I agree to add it to known issues section.
Hi Kevin,
I’m looking to upgrade to SCOM 2022 from 2019 with SQL Server 2016. Do I need to upgrade the SQL Db first to be able to do an inplace upgrade?
Kind regards
Chris
Sorry, ignore that I read your message about SQL above. Yes I do need to upgrade SQL first 🙂
i am getting the following error when running custom reports after upgrading from scom 2019 to 2022
Note: The following information was gathered when the operation was attempted. The information may appear cryptic but provides context for the error. The application will continue to run.
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.EnterpriseManagement.Mom.Internal.UI.Console.ReportForm.RunReportJob(Object sender, ConsoleJobEventArgs args)
Hi Kevin, we recently upgraded from SCOM 2019 to 2022 and we are seeing some NTLM authentication coming from our SCOM SSRS server to the DefaultSDKServiceMachine Management server.
It seem to happen at regular cycle, once every 12hrs or if we reset the SQL Reporting Services.
All the other auth are using Kerberos and we are not getting any SPN errors.
rsreportserver.config contains the option already.
I saw the release note that SCOM 2022 now support NTLM hardened system but no details or anything more.
This page is not really helping either
https://learn.microsoft.com/en-us/system-center/scom/deploy-install-reporting-server?view=sc-om-2022#install-reporting-services-on-ntlm-hardened-enterprises
We did an upgrade from 2019 to 2022 on the reporting server and never saw any option for authentication.
Any idea on what could cause this?
In step 8 you say “Optimize Registry settings for management servers”, can SCOM 2019 UR4 run with these settings without issue?
I have this post in the System Center Q&A, but thought I’d post here as well.
Kevin, when you say “TLS 1.2 enforced on management servers or SQL but missing prerequisites” in What to do when things go wrong?, does this mean both the OS and OM registry keys mentioned here – https://learn.microsoft.com/en-us/system-center/scom/plan-security-tls12-config?view=sc-om-2022&viewFallbackFrom=sc-om-2022How%20to%20implement%20Transport%20Layer%20Security%201.2
need to exist on the MSs and SQL servers before upgrading. I also have seen one comment that suggests they need to be on every agent server as well in order for the agents to update from pending management correctly – is this true?
I did an upgrade from SCOM 2019 UR4 to SCOM 2022, SQL on a separate server, everything seems to have gone well except that in the administration view under databases my Operations Manager Data Warehouse is still showing the version to be 10.19.10569.0 whereas the Operations Manager DB is showing 10.22.10118.0. This is disconcerting, so I ran select * from SqlPatchVersion on both databases and confirmed the DW is only on 10.19.10569.0. Is there a way to fix? I’m guessing I need to start over
🙁
That’s interesting. You might consider opening a support case…. I checked with support and it is not a well known issue to them. I’d pull the logs from the upgrade and see if the DW failed for some reason, or if the DW is huge, if the script to upgrade it is still possibly running and not completed. You can run the DW upgrade script manually…. I believe you just call Datawarehouse.OM12TH.Upgrade.sql which is in the setup media.
Kevin, we have SCOM 2019UR4 with SQL2019 CU8
Our database ingeneer insists to upgrade SQL to 2022 but we haven’t planned to upgrade to SCOM 2022 yet
Does SCOM2019 UR4 support SQL2022 as a database server ?
SQL requirements for SCOM 2019 are clearly documented: https://learn.microsoft.com/en-us/system-center/scom/plan-sqlserver-design?view=sc-om-2019#sql-server-requirements
If your DBA insists on upgrading SQL to a level not supported by the application, I’d question the sanity of that approach.
Hello Kevin,
Thank you for the guide; its be incredibly helpful. I am however running into an issue when upgrading the management servers in my environment. We’re running WS2019, and SQL2019. The first management server went fine upgraded the Operations Database, and the DataWarehouse, Management Server etc. 2 Other management servers completed successfully, however on my last 2 Management Servers I am getting the same error;
[12:01:08]: Error: :LaunchMSI: MSI C:\Temp\Microsoft – System Center Operations Manager 2022\Setup\AMD64\Server\OMServer.msi returned error 1603
[12:01:08]: Error: :ProcessInstalls: Install Item Management Server failed to install. We did not launch the post process delegate.
[12:01:08]: Always: :SetErrorType: Setting VitalFailure. currentInstallItem: Management Server
[12:01:08]: Info: :SetProgressScreen: FinishMinorStep.
[12:01:08]: Always: :!***** Installing: OMDATAWAREHOUSE ***
[12:01:08]: Info: :ProcessInstalls: Rollback is set and we are not doing an uninstall so we will stop processing installs
Then the installation rolls back and fails. Leaving my server in a position where it needs to be restored from backups (or so it seems, as it uninstalls the previous management server, and places the system in a place where the setup wizard thinks the management server is actually installed.)
Any tips? Or should I create a support case?
I have same issue, only way to restore from last backup, same design UAT worked fine but failed in Prod.
[11:40:11]: Always: :LaunchMsi: Launching D:\SCOM_2022\Ops_Mgr_2022\Setup\AMD64\Server\OMServer.msi
[11:40:54]: Always: :MsiInstallProduct finished for msi D:\SCOM_2022\Ops_Mgr_2022\Setup\AMD64\Server\OMServer.msi.
[11:40:54]: Always: :LaunchMSI: Setting rollback to true
[11:40:54]: Error: :LaunchMSI: MSI D:\SCOM_2022\Ops_Mgr_2022\Setup\AMD64\Server\OMServer.msi returned error 1603
[11:40:54]: Error: :ProcessInstalls: Install Item Management Server failed to install. We did not launch the post process delegate.
So, I finally upgraded my test instance to SCOM 2022, and all went perfect using this checklist. However two issues: The gateway upgrade didn’t prompt for a file location, and removed my G:\program files… installation and installed the Gateway files and folders to c:\program files… not a big deal, but I liked having in on G: rather than C:
Issue #2 – I find that updating the agents from Pending Management installs the APM option even though the existing installation of the agent (SCOM 2019 UR4) has been installed with the NOAPM switch. We don’t install the APM component as we were burnt with the 2016 version and all it’s issues. Don’t really want to risk it again…
How do I get my agents upgraded without getting the APM option installed?
Advisor rule
what is advisor for ? can i remove all of them?
i have a last question how can i see the content of a rule please
for example microsoft.systemcenter.advisor.mpupdate or
microsoft.systemcenter.advisor.syncalert
thanks
Hi Kevin,
I’m trying to upgrade a SCOM 2019 environment at one of my customers but it keeps failing. (I’ve done some upgrades at other customers already without issues)
The management server gets uninstalled, but when starting the upgrade of the OperationsManager database it fails with this error:
Error: :Sql error: 16. Error: 7642. Error Message: A full-text catalog named ‘FullTextCatalog’ already exists in this database. Use a different name.
I even tried (after DB + management server restore) to drop that full-text catalog but then the wizard complains about the full-text search not being installed. So that doesn’t fix it either :).
Is that something you came across already?
Best regards,
Bert
We had the same issue today and noticed (when trying to do a new install on the management server instead) that the Full Text Search feature was missing on the SQL servers. The process of restoring the database and management server is still underway as of now, but this might be the case with your customer’s environment. In our case it happened because we migrated the databases to new SQL instances. The weird thing though is that SCOM didn’t show any alerts or issues about this missing component the last few months, as if this feature is only required when installing/upgrading.
Hi,
After upgrade we receive these alerts:
The INSERT statement conflicted with the FOREIGN KEY constraint “FK_State_BaseManagedEntity”. The conflict occurred in database “OperationsManager”, table “dbo.BaseManagedEntity”, column ‘BaseManagedEntityId’.
The statement has been terminated…
Management Configuration Service group failed to perform dirty notification work item for a period of time
Management Configuration Service group failed to perform delta synchronization work item for a period of time. Last error message (if available): Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.ConfigManager.ConfigurationChangeException: Configuration change occurred during delta synchronization. Delta will be restarted
Management Configuration Service group failed to perform Configuration Store index optimization work item for a period of time. Last error message (if available):
Checked event ids, event 10570 and 29181 is triggered.
Any suggestions?
Event id: 29181:
OpsMgr Management Configuration Service failed to execute ‘SnapshotSynchronization’ engine work item due to the following exception
Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetConfigurationSnapshotNextBatch(SnapshotProcessWatermark watermark)
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetConfigurationSnapshotNextBatch(SnapshotProcessWatermark watermark)
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.SnapshotSynchronizationWorkItem.TransferData(SnapshotProcessWatermark initialWatermark)
at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.SnapshotSynchronizationWorkItem.ExecuteSharedWorkItem()
at Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.SharedWorkItem.ExecuteWorkItem()
at Microsoft.EnterpriseManagement.ManagementConfiguration.Interop.ConfigServiceEngineWorkItem.Execute()
———————————–
System.IndexOutOfRangeException: SizeNumeric_317C3106_D8A2_BE95_1B12_4DB8E13879FB
at System.Data.ProviderBase.FieldNameLookup.GetOrdinal(String fieldName)
at System.Data.SqlClient.SqlDataReader.GetOrdinal(String name)
at System.Data.SqlClient.SqlDataReader.get_Item(String name)
at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.PropertyDictionaryBuilder.BuildPropertyDictionary(IDataReader managedEntityReader)
at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagedEntitySnapshotReadOperation.ReadManagedEntity(SqlDataReader reader)
at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagedEntitySnapshotReadOperation.ReadData(SqlDataReader reader)
at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)
Hi Kevin,
I’m upgrading a customers environment and I get a failure when upgrading the database, going through the logs I see this error:
Error: :Sql error: 16. Error: 10345. Error Message: The assembly hash ‘0xEC312664052DE020D0F9631110AFB4DCDF14F477293E1C5DE8C42D3265F543C92FCF8BC1648FC28E9A0731B3E491BCF1D4A8EB838ED9F0B24AE19057BDDBF6EC’ is already trusted.
I see a few of the same error. I would expect that it would continue as this assembly is already trusted but apparently it fails on that.
After restoring the Database and the management server I tried to upgrade SCOM runs fine again so no harm done.
Is this something you have seen before?
Best regards,
Bert
I am having same issue but restoring database or MS is not solution, any reason why we get this error
Looks like it is bug in SCOM upgrade, when assembly is already trusted, it should ignore and go to next steps but it tries to drop the table to create new table to trust the same assembly hash and fails.
Run into this same issue as well. Solution was to manually remove the two assemblies from the database just before the upgrade.
To check the assemblies:
select * from sys.trusted_assemblies
To remove the assemblies:
EXEC SYS.sp_drop_trusted_assembly (do this for both)
After the upgrade the assemblies are back in place.
Step 16:
I resolved this by using these PS cmdlets
Get-SCOMNotificationSubscription |Disable-SCOMNotificationSubscription
maybe you would like to add this step?
Verify “servers with web console” to have HTTP binding defined
I had an upgrade error on my first mgmt server.
Webconsole upgrade failed.
Looking in the log:
Error: :Inner Exception.Type: System.ApplicationException, Exception Error Code: 0x80131604, Exception.Message: Failed to find a port that uses the http protocol.
I only have https bindings defined, no http, as I like to have my environment secure.
When you upgrade a management server, is it important to ensure the SCOM services are still stopped / disabled? Or do these need to be running on a Management Server that is currently going through the upgrade process?
I don’t recommend leaving them stopped. If you follow my guide, I never recommend disabling anything. Additionally, I recommend rebooting the MS before upgrade, which will have all the services running post reboot.
Thanks Kevin. I have read elsewhere that when upgrading the first management server, the others in the resource pool should have their SCOM services stopped? And in your guide, if someone chose not to optionally reboot the Management Servers, services would still be stopped, from the Database backup (Step 20). To confirm…. you would recommend running the SCOM 2022 Setup on the 1st Management Server, while other management servers are still running workflows and agent connections?
Yes. I know in years and versions past, some people recommended stopping services. I am not sure why. I only stop them to get a consistent state for the backup. During the upgrade the services need to stop and start multiple times, and potentially connect to other management servers.
I am having issue with upgrading from SCOM 2019 UR5 to SCOM 2022. In setup log I have this error:
Found error in 2|Microsoft.SystemCenter.ManagementPack.Recommendations|10.19.10304.1|Microsoft.SystemCenter.ManagementPack.Recommendations|| with message:
Version 10.22.10034.0 of the management pack is not upgrade compatible with older version 10.19.10304.1. Compatibility check failed with 1 errors.
Would just removing Management Pack Recommendations help with the upgrade?
Thanks
To further add to this, in next line there is also this error:
ClassType: LW.Microsoft.3rdParty.Dell.EMC exists in current version 10.19.10304.1 of the management pack but does not exist in new version
Hopefully just removing Management Pack Recommendations can help with this error.