KB Article for OpsMgr
New Features
Download from Catalog
Download for the NEW Simplified Management Server Update
Download Updated UNIX/Linux Management Packs
Recommended hotfix page
New Features:
- Change tracking for management packs
- Improvements in scheduled maintenance mode
- Favorite reports in web console
- Support for folders in monitoring view of web console
- Support for CentOS 8
Key fixes:
- See the KB article. There are too many fixes to list here!
NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2019 and never applied an update rollup – you can go straight to the latest one available.
Let’s get started
From reading the KB article – the order of operations is:
- Install the update rollup package on the following server infrastructure:
- Management Servers
- Web Console Servers
- Gateway Servers
- Operations Console Servers
- Reporting Server
- Apply Agent Updates
- Update Unix/Linux MP’s and Agents
Management Servers
Updates in SCOM 2019 have CHANGED. There is a new process for updating management servers that differs from previous versions of SCOM. Download the single file management server update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates.
- Download: LINK
It doesn’t matter which management server I start with. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I have 2 management servers My first management server holds 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.
Notice the new EXE file, and a MSP file exist for the Server update. The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching. I will ONLY demonstrate and recommend the EXE file.
Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.
- ***Note: One of the changes in SCOM 2019 Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL. The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases. You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.
My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.
I will start with KB4558752-AMD64-Server.exe
This is a self-extracting executable, that kicks off a simple update tool. Accept the license terms, and click “Install”
This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.
If you have an issue – you can review the setup logs:
- Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
- SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
- MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log
Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update: KB4558752-AMD64-WebConsole.msp
Lastly – install the Console Update (make sure your console is closed): KB4558752-AMD64-Console.msp
You can reboot the server at this time if you were prompted to in order to complete the update. If you were not prompted to, you do not need to.
Additional Management Servers:
Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers. You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.
You can use the same EXE file you used for the first management server. The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if needed.
Updating Gateways:
Open an elevated command prompt, and run the update: KB4558752-AMD64-Gateway.msp
The update launches a UI and quickly finishes.
I can spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:
You can delete any older UR files that might be present. We don’t clean these up automatically.
Updating Reporting:
On your server that hosts the SCOM Reporting role, run the update: KB4558752-AMD64-Reporting.msp
Update Agents:
Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):
***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin. The Agents MUST have “Remotely Manageable” set to “Yes”.
You can approve these – which will result in a success or failure message once complete:
Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:
And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery. This has long been a pain point in SCOM:
https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
Update UNIX/Linux MPs and Agents:
You can get the current Unix/Linux MP updates HERE.
Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor. Here is the FULL list:
In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:
This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.
Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits. If they are not present, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.
After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:
Now you can deploy the Linux agent updates:
If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:
Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.
Verifying the update
There are new views in the SCOM console to help with this and make this process MUCH easier. You do need to wait long enough for the discoveries to run in order for these to update the views.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known Issues:
1. The Web Console views may not work (500 error) for scoped users who are not SCOM Admins. These same views work fine for SCOM Admins.
- This primarily affects Web Console roles installed using Network Authentication. This does not appear to impact Web Console servers installed using Windows Authentication.
- On the web console server – find the folder: \Program Files\Microsoft System Center\Operations Manager\WebConsole\MonitoringView\TempImages
- Change the security on this folder for Authenticated Users – and add “Write” capability:
2. Once you apply UR2, you cannot install, or re-install SCOM reporting Role.
- You will see an error in the setup UI (when you supply a management server name) that states “Unable to connect to the Data Access service for this management server. Ensure the Data Access service is running and that the service, the management group, and setup are all the same version.”
- In the setup log, located at C:\Users\<username>\AppData\Local\SCOM\LOGS\OpsMgrSetupWizard.log – the last line will appear similar to: Error:The management server is a different version than the current setup build. Please use a different management server or the correct version of setup. Server Version: 10.19.10311.0
- SCOM Reporting Setup looks for a very specific value in the SQL database, which has been updated by the UR1 and setup now rejects continuing.
- Apply the following workaround to install/reinstall SCOM Reporting role:
- QUERY the OPERATIONSMANAGER database, and record the VERSION number that is returned. You will need this value later. You need to change the PrincipalName to your SCOM Management server that you point the reporting install to.
-- 10.19.10050.0 - 2019 RTM -- 10.19.10311.0 - 2019 UR1 -- 10.19.10349.0 - 2019 UR1 with post UR1 Hotfix -- 10.19.10407.0 - 2019 UR2 USE OperationsManager SELECT PrincipalName, Version FROM MTV_HealthService WHERE IsManagementServer = 1 AND PrincipalName = 'OMMS1.opsmgr.net'
UPDATE the VERSION entry in the OpsDB to match the RTM version number which is 10.19.10050.0 just for this management server.
UPDATE MTV_HealthService SET Version = '10.19.10050.0' -- 2019 RTM WHERE PrincipalName = 'OMMS1.opsmgr.net'
Install SCOM 2019 Reporting, and choose this same Management Server. Reporting install will work now.
REVERT the VERSION entry in the OpsDB to match the original value you recorded
UPDATE MTV_HealthService SET Version = '10.19.10407.0' -- 2019 UR2 WHERE PrincipalName = 'OMMS1.opsmgr.net'
Done!
Pingback:SCOM 2019 UR2 released | TopQore Blog
Thanks a lot. How can I update “Audit Collection Servers”? After apply UR2 the version continue 10.19.10050.0.
UR2 does not contain any updates for ACS
I went through the UR2 update and no errors encountered, but curiously the ‘Databases’ section under “Administration > OpsMgr Products > Databases” is empty. I upgraded the agents on both Ops and DW databases to UR2, and all other roles report correctly as UR2.
I don’t see any errors/alerts in the OpsMgr anywhere. The OpsMgr environment otherwise appears to be functioning without issue. Is it possible this is a UR2 specific issue?
This uses a discovery on the SQL server, it is likely just something about your SQL connection string that is causing our discovery not to work. I would not worry with it.
Good to know. And now the DB information is showing up in the console. I ended up rebooting each Management server for an unrelated reason (the UR2 installs didn’t prompt for restarts). No idea if it was that or just a timing issue, but regardless looks to be working now. Thank you sincerely for your efforts documenting all things SCOM. I’m working on a 2012R2-to-2019 “migration” and the info regarding 2019 and the SQL RunAs stuff is fantastic. I’m extremely grateful having this site as a resource.
Morning, Can I use the SCOM 2019 Admin Console to work with my SCOM 2012R2 environment? Two other admins have access to the SCOM 2012R2 Admin Console console but via Citrix. Would be great if I could just replace this with the SCOM 2019 Admin Console for both..
Morning, I posted a question but after the screen refreshing it does not show up.
Question: Can I use the SCOM 2019 Admin Console to connect to my SCOM 2012R2 environment, instead of having to have two admin consoles installed? I RDP to the servers, so not a big issue. But my other two admins access it via Citrix.
Thank you. TS
It can – but Microsoft does not support it. There is no interop testing of SCOM 2012R2 and SCOM 2019. Only SCOM 2016 and SCOM 2019.
I just updated our SCOM test system and found a strange behaviour: I started updating my agents that were in “Pending Management” by selecting a bunch of entried clicking on “Approve”. This opens a new window to enter the credentials as usual, and then the new window displayed the servers where the agent was being updated with status “Started”. So far so good.
Then I switched to “Monitoring” and at the same time the status window was closed. So I had no chance to see in this windows whether the agent updated succeeded or not.
Yes, SCOM has always done that. You can use the task status view after that.
Issue:
After updated to UR2, under “Program and Features” the version information from the Microsoft System Center Operations Manager will not be updated to 10.19.10407.0.
Kevin, your site has always helped me, and I thank you for it.
Question on this UR2;
After running through the update on both the Management Server, and SQL servers I did a check of the version. The server, console and the SQL DB show the new version, but when I run the PS Get-SCOMManagementGroup the Version still shows the old value.
Is this a known item?
I was able to answer my own question; as the call looks at the MP version and not the Application Version.
(Get-SCOMManagementServer -Name $MSServerName).Version
I updated our test environment and everything worked fine. But I checked the installed version and the Reporting Server is still UR1 (the others components are ok). During the upgrade I had no errors and there are no new LOGs in C: \Users\\ AppData\Local\SCOM\ LOGS
Thank you, B.
resolved “problem” with update UR2 Report: I generated the log with the /l switch and now it’s ok….
I am Receiving this error going from RTM to UR2, “An error occured while configuring the OperationsManager Database”. In the log, I have the following:
DEBUG: Error 2769: Custom Action _InsertAgentsIntoPendingUpdate_OM12_Mode.1451A536_2C9B_42F2_A37A_C9C6460E7EEA did not close 13 MSIHANDLEs.
Internal Error 2769. _InsertAgentsIntoPendingUpdate_OM12_Mode.1451A536_2C9B_42F2_A37A_C9C6460E7EEA, 13
MSI (s) (C0:7C) [16:58:01:025]: NOTE: custom action _InsertAgentsIntoPendingUpdate_OM12_Mode.1451A536_2C9B_42F2_A37A_C9C6460E7EEA unexpectedly closed the hInstall handle (type MSIHANDLE) provided to it. The custom action should be fixed to not close that handle.
I have reinstalled SCOM RTM clean and tried the upgrade right away, also tried reinstalling on clean VMs.
That’s odd. There was a change made to help this process work even if you did not have sysadmin to the SQL server. I am assuming you did not run the UR2 package as someone with Sysadmin? Were you using the EXE patch or the MSP server update patch when you got this?
I ran the “KB4558752-AMD64-Server.exe” file. I thought it may be a SQL permission issue, however I am sysadmin on the SQL server. I also added dbo to my login on the operationsmanager and datawarehouse databases.
Is there is an alternate method or workaround I am happy to perform the upgrade any way needed to get it done. I have full reign on the application and SQL servers.
Can you email me a copy of the full log. I tried to reply to your email in your post and it bounced.
Hi Kevin, I sent you a message, I did not see your email address listed.
Hello!
We have the same troubles. Management Server 1 runs without troubles and on the 2nd Mgmgt Server there was the same Error like Jason Seymore sent. Have you a Solution for that? Regards!
I have applied UR2 successfully.
But when I import management pack, it shows “import failed”.
Windows Server Operating System Reports could not be imported.
If any management packs in the Import list are dependent on this management pack, the installation of the dependent management packs will fail.
Database error. MPInfra_p_ManagementPackInstall failed with exception:
[SQL Error Code: -2146232060][MP ID: 57f7eaa0-2430-85d9-05a4-4a897c9d4d9b][MP Version: 10.1.0.2][MP PKT: 31bf3856ad364e35] Procedure or function p_MPImportXML has too many arguments specified.
Rerun the update-rollup_mom_db.sql in the OperationsManager Database.
The SQL script for UR2 will be located at C:\Program Files\Microsoft System Center\Operations Manager\Server\SQL Script for Update Rollups
https://docs.microsoft.com/en-us/answers/questions/72068/scom-2019-ur2-management-packs-cannot-be-imported.html
Carl,
Not sure if it helps you or not, I had the same error, when I was verifying the management server versions, I noticed my database version wasn’t updated. I had to re-install the update with the EXE file I had used windows updates the first time around.
Yep, Windows Updates will not update the database. If you use that, you must run the SQL scripts manually, the old-school UR update method.
Kevin,
Thanks for your blog, it is always helpful when performing these updates.
Where can I get the SQL Scripts and MP for applying manually?
Thanks Kelvin for your inputs,
Christopher,
I checked the database version is 10.19.10050 under Administration –> Operations Manager Products –> Databases.
I think it should be UR2 already.
Thanks.
Carl.
Christopher,
Many thanks. I download UR2 setup file and applied it. The database version changes to UR2 now.
I can import MP now.
Thanks.
Carl.
Hello Carl, Kevin,
I also downloaded the Setup Files and run the below SQL files. It shows updated SQLPatchVersion, but from Management Console –> ADministration –> Operation Management Products –> Databases, it still shows older version.
UR_DataWarehouse.sql – OMDW database
update_rollup_mom_db.sql – OM Database
is there any changes needed in applying SQL patches, or any restarts required.. please suggest.
You generally should not run this manually. You should re-run the UR2 update (using the EXE is preferred) and that will import the management pack that handles the SQL updates.
Thank you for reply …
When we run the setup through, EXE even with Elevated rights, the Setup hangs at “Removing Backup Files” .. logs also shows the same .. We tired 3 times and same result, it stays there forever, Unless we cancel it.
when we run using the MSP, it hangs at 59 Seconds remaining .. and logs wit same message “Removing backup files.”
I recently updated to UR2, with Kevin’s assistance, however in my struggles to complete the installation and upgrade, I completely forgot to install the reporting server, and receive error “ensure the data access service is running and that the service, management group, and setup are all the same version” which is of course not the case, everything is UR2, and I am trying t install RTM reporting services.
Is there a way to either install the UR2 version or reporting services without the underlying version, or am I going to have to remove everything and install the base version again then re-apply UR2?
This is covered in this same article – see #1 in “Known issues” with the workaround.
Thank you sir! a little ‘rtfm’ would have been helpful on my part in this case. Your articles are amazing, keep up the great work!
Hello, UR2 packages are not found in Microsoft Update Catalog… I can not download it!!!
I have confirmed and reported this – stand by.
Thks Kevin for help me, It was driving me crazy ….
I believe this might be because they get rebranded at some point from optional to recommended…. and this causes them to disappear for a day. Will have to check back on monday/tuesday
These are still missing, We did not find any results for “4558752” when trying to download the stand alone elements of the KB?
Yes – confirmed – this is a side effect of us changing these updates from Optional to Recommended in Windows Update. They should be available soon.
Appears!!! Thks
If I have KB4558752-AMD64-Server.exe. Is that all I need to run the update, or do I need to download the msp files as well? Sorry, I am a little confused if the exe is capable of doing the installs or if it unpacks into the msp files and I have to run them as well.
This link (http://www.catalog.update.microsoft.com/Search.aspx?q=4558752) does not return anything from the catalog.
Thank you.
No – see my examples above. The EXE is good to run:
1. The Server role update.
2. The SQL scripts
3. The MP import.
The EXE does NOT contain updates for:
1. Web Console Role
2. Console Role
3. Gateway Role
These must be downloaded separately. Just wait a day or two and everything should be back on the catalog.
Running the UR2 server.exe file on my management servers ends with a reboot, which I can’t decide myself if I want to run. It simply just reboots the management server.
Is this normal procedure in 2019 I didn’t see that in 2016?
No – that’s not normal. However – I also had that happen with a customer, and could not repro it in my lab. I did report it to the Product group. I will send them another report of this in the wild.
This happens to me on 2 of my 8 MGMT servers (UR1 did the same) no question just reboots on its own? All 8 Servers are the same OS/patch level and in different data centers so cant be related to VMWare etc.
Management Servers in different datacenters?
Ewwww. Is that for some kind of Disaster Recovery design?
Yep for DR, both Data centers are based in UK so latency between the two is around 3ms.
Hi Kevin, when you say “The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases.” which RunAs account are you refer to.
2nd question, I am installing scom 2019 and want to take advantage of GMSA, is there any way to install a fresh SCOM with UR2 included so that i can use GMSA from the start?
Thank you
Hello,
I have just upgraded to UR2, and everything completed with no errors. Everything under “Operations Manager Products” is showing accurate information, UR2 and the version looks good. So far everything is working fine. The one issue I am having is.
When I navigate to the built-in HTML 5 dashboards they all show this error in any of the widgets: “The required anti-forgery cookie “__RequestVerificationToken_L09wZXJhdGlvbnNNYW5hZ2Vy0″ is not present.”
This is only happening in Chromium browsers, it seems to be working fine in IE. Has anyone else experienced this?
ANY widget/dashboard – or specifically in the SQL management pack? The SQL MP dashboard was updated with a fix for this.
Did you update the Management Packs and SQL script updates using the Server EXE update: KB4558752-AMD64-Server.exe or did you use some other method for the server update, such as MSP file or Windows Update?
What version is your Microsoft.SystemCenter.HtmlDashboard.Library MP? It should be 10.19.10407.0
Thank you so much for your quick reply.
I feel a little silly, but while validating some of the information you replied with, I realized I just needed to clear my browser cache/cookies and now all is working well.
Your posts are always super helpful, and this one worked perfectly for me.
Thanks again!
Kevin, have you seen any reports of the Reporting “Update Installed” staying as UR1/KB4533415 and “version” as 10.19.10311.0 when everything else has updated fine to UR2 (MGMT servers, consoles, web servers, MPs etc.)
I have run the KB as Administrator from command line, it dos not take long at all, so not sure if it is not completing but no errors?
No – I haven’t seen that. Did you bounce the SCOM Agent on the Reporting server? Is the SCOM agent healthy?
Yes I bounced the agent and its healthy after the update… I have just checked this morning and it has now updated the version in the console, seems it took hours for it to do this….
I had the same issue, but rebooted my Reporting Server and the console now shows the correct version.
Hi Kevin,
We applied UR2 and everything went smooth. But the below event keeps writing every 10mins in the Management Server OpsMgr Event Logs and “Data Warehouse Database Component Deployment Recovery State” : “Data Warehouse failed to deploy database component” alert triggered in the console. Any clue?
Event ID: 31565
Failed to deploy Data Warehouse component. The operation will be retried. Exception ‘DeploymentException’: Failed to perform Data Warehouse component deployment operation: Install; Component: Script, Id: ‘bf47fb9d-6e24-5594-1830-1d4d8dac3e48’, Management Pack Version-dependent Id: ‘b868ade9-4b41-92c1-d2af-37b8d7801414’; Target: Database, Server name: ‘XXXXXX\XYZ’, Database name: ‘OperationsManagerDW’. Sql execution failed. Error 777971002, Level 16, State 1, Procedure DeploymentOperationValidate, Line 279, Message: Sql execution failed. Error 777971104, Level 16, State 1, Procedure DeploymentOperationValidate, Line 171, Message: Deployment operation validation failed. Component alrady installed. Operation: ”Install/Upgrade”; Component type: ”Script”; Component id: ”BF47FB9D-6E24-5594-1830-1D4D8DAC3E48”; Management pack version: ”B868ADE9-4B41-92C1-D2AF-37B8D7801414”; Target: ”Server name: XXXXXX\XYZ; Database name: OperationsManagerDW; Dataset ID: [null]” One or more workflows were affected by this. Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Component Instance name: Data Warehouse Synchronization Service Instance ID: {82730503-D691-50D7-1697-D8CE9E9B81E7} Management group: MG04
Is this the situation with a shared data warehouse?
Hi Kevin,
Yes, it is with shared data warehouse.
any solution to this issue. for me this error is still continue to come. I did UR2 2019 update a month ago. I have clustered SQL instance of DW.
I have the same issue,can you please get back to us if you could solve it ?
Hi Kevin,
Yes, it is with shared data warehouse.
My guess is, that we didn’t not anticipate nor test with this scenario. I’d recommend bouncing your management servers, and your SQL servers, and see if that kills the process attempting the update. If it does not, I’d open a support case with Microsoft.
In the future, I’d not use the EXE updater for subsequent UR’s, as it will attempt SQL script updates to the warehouse from each management group. I’d perform the MSP update, then to the SQL script and MP imports manually to avoid this issue.
We should raise a bug (via the support case) to handle the update of the DW from a shared management group, or ignore if the DW is already updated.
Hi Kevin,
We tried another MG with old process. (Manual Execution) – We have not updated UR2 (with exe) anytime before on this MG.
Everything went smooth, but while we try to execute the DB Query, Got the below Error. Can we ignore this error?
Msg 2714, Level 16, State 5, Line 3934
There is already an object named ‘FK_MaintenanceModeStatus_BaseManagedEntity’ in the database.
Msg 1750, Level 16, State 1, Line 3934
Could not create constraint or index. See previous errors.
Hey Kevin, We have the same issue with the “Component alrady installed” message but we do not have a shared data warehouse as we only have a single Management group. Is there a hotfix on the way or any workaround to limit the error message?
Thanks
Any idea what happened that might have caused that? We have a couple customer cases of this, but I am not sure how to repro just yet.
Simply installed the UR2 with the .exe on my management servers as the previous poster did. We have 4 MS accross 2 sites. With 2 in each site. This error only shows on the Management servers in a remote site. The SQL server they are all pointing to has the opsdb as well as the opsdw.
Hi Kevin,
We tried another MG with old process. (Manual Execution)
Everything went smooth, but while we try to execute the DB Query, Got the below Error. Can we ignore this error?
Msg 2714, Level 16, State 5, Line 3934
There is already an object named ‘FK_MaintenanceModeStatus_BaseManagedEntity’ in the database.
Msg 1750, Level 16, State 1, Line 3934
Could not create constraint or index. See previous errors.
The same here:
FK_MaintenanceModeStatus_BaseManagedEntity
And issue we can not anymore install Linux agents. When Clicking on the UNIX/LINUX Computers SCOM comes with this error:
An item with the same key has already been added
Application Version: 10.19.10407.0
Severity: Error
Message:
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at Microsoft.SystemCenter.CrossPlatform.UI.OM.Integration.MonitoringObjectPathToMonitoringObjectDictionary..ctor(IEnumerable`1 monitoringObjects)
This is not really as much related to UR2, as it is the fact you are discovering a Linux system with two different MP’s. This happens when you discover RHEL or SLES commonly with an older MP, AND you are using the newer Agnostic “Universal Linux Monitoring” mp. See: https://social.technet.microsoft.com/Forums/systemcenter/en-US/5a4be487-9472-4be0-afe6-df7bc7168465/scom-2019-universal-linux-monitoring-mp-duplicate-items?forum=operationsmanagermgmtpacks#3ae44fc7-aef5-4e43-849f-f837068a845f
Hi Kevin,
We have raised MS Case and they confirmed other customer also faced same issue and the workaround is to comment out that part of the script and run again.
There is also one more point to note:
We never imported any MP during our UR2 update (Executed the MSP Package, not the exe), but all the MPs already imported, when we tried to import in the last step. Not sure KB4558752-AMD64-Server.msp package has included to import and execute the SQL Scripts.Same has been communicated to the MS Engineer.
Hi Kevin,
MS Update: They have tested in LAB and confirmed.
“From this behavior I can say it’s safe to determine that the new .msp files used to update to UR2 automatically import the MPs, AND automatically run the SQL script.”
Hi Kevin
Since upgrading to UR2 we have been unable to push agents, get the “there were no computers discovered” error we have added all the run as permissions etc? I can install the agent manually have tried all the troubleshooting steps and no luck.
Hi Kevin,
Thank you for your great support via your blog, it has helped us a lot already.
We ran into some problems updating our SCOM2019 Environment to UR2 (skipping UR1).
The installation went smooth on every server and component but shortly after the update a lot of alerts started showing up in the console(SCOM Agent Max Objects Limit Reached) indicating that our ManagementServers are not properly accepting/forwarding the data.
Upon investigating the issue we found out that on two of our three ManagementServers the “MonitoringHost.exe” was crashing every ~2 minutes which is shown by Events in the Windows/Application log ID 1000:
Faulting application name: MonitoringHost.exe, version: 10.19.10014.0, time stamp:
Faulting module name: MSVCR120.dll, version: 12.00.40660.0, time stamp:
Exception code: 0xc0000409
Faulting application path: C:\Program Files\Microsoft System Center\Operations Manager\Server\MonitoringHost.exe
Faulting module path: C:\Windows\System32\MSVCR120.dll
Sadly we could not find more usefull information as to why the application crashes.
What we have tried already:
Reverting MS2 and MS3 to a snapshot before the update and then:
1)Installing UR1 (everything works fine) and afterwards installing UR2 -> Problem again
2)Same as 1) but using the Server.msp file instead of the Server.exe
3)Update to UR2 then repairing the installation via the Setup.exe
4)Rebooting every Server before and after the Update
Our Environment (Windows Server 2019):
All ManagementServers hold only the MS Role and have the OperationsConsole installed
OpsDB – UR2
DWHDB – UR2
MS1 – UR2 (No Problem)
MS2 & MS3 – Currently on UR1 since it seems to be working fine
Multiple GWs – UR2
Web/Reporting – UR2
Do you have any idea what might cause the Monitoring.exe to crash or have other possible solutions that we could try in oder to fix the problem?
I have not seen this – and I recommend you open a Microsoft Support case for this issue.
Hi, a quick update:
We found out that the affected ManagementServers were only the ones that were in the Unix/Linux-Resource Pool but could not fix the issue ourselfs and opened up Support Case like you suggested.
After investigating Microsoft told us to replace the MOMWsManModules.dll (Located in C:\Program Files\Microsoft System Center\Operations Manager\Server) with the dll file from a server with UR1 installed and this workaround fixed the issue for us.
So the MOMWsManModules.dll version 10.19.10140.0 instead of version 10.19.10153.0 did the trick for us.
We still do not know what the root issue was nor do we know how this will affect us in future updates but at least do we have a stable environment on UR2 at the moment.
Hi Kevin,
We just installed the UR2 to the environment successfully, updated some agents, but the on the agent’s side, the registry, and at the agent’s properties, still shows, that the installed product version is 10.19.10014.0.
The agent update was also successful, but the version is still remains the same, after server reboot. At the SCOM Console, the agent is shown with the UR2 installed.
Is it a known issue or can we do anything?
I have the same “issue”. In registry still show version 10.19.10014.0.
In installed directory HealthService.exe is in newest version number 10.19.10153.0
Hello Kevin,
After upgrading from SCOM 2019 RTM to UR2 I get this event error:
Mom failed executing SQL Job p_ScheduledJobsEveryFiveMinutes. Details: The EXECUTE permission was denied on the object ‘sp_add_job’, database ‘msdb’, schema ‘dbo’.
The EXECUTE permission was denied on the object ‘sp_add_jobschedule’, database ‘msdb’, schema ‘dbo’.
The EXECUTE permission was denied on the object ‘sp_add_jobserver’, database ‘msdb’, schema ‘dbo’.
The EXECUTE permission was denied on the object ‘sp_help_jobactivity’, database ‘msdb’, schema ‘dbo’.
The data access service account or management service action account might not have the required permissions (SQLAgentOperatorRole) on MSDB database
The Schedule is invalid.
Did you come across this error as well? This error happens when I create a Schedule Maintenance and event occurs every 5 minutes but the Schedule Maintenance works.
Before I change the persmission I would like you opinion what I should do?
THx.
Arno
UR2 should not cause this. Your SQL permissions are missing for the accounts to the MSDB database. Please see https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/
Hi Kevin,
The permissions for the MGMT Server Action Account were missing on the following DB. The error event in Ops. Manager disappeared.
MSDB:
public
SQLAgentOperatorRole
SQLAgentReaderRole
SQLAgentUserRole
Funny, I never really changed anything on the permissions. Anyway, thanks a lot for your help.
Cheers,
Arno
By the way, and this just started after the upgrade to UR 2. I checked.
Hey,
if i run the KB4558752-AMD64-Server.exe on my Management Server, the Update will hung at Management Server “Removing Backup Files” forever. In the log file the last line is
12:35:39]: Info: :Install Progress – (RollbackCleanup) Removing backup files
The User, which runs the .exe ist Administrator.
Can you help please?
Hi,
Got exactly the same issue. Then stumbled upon this one: https://social.technet.microsoft.com/wiki/contents/articles/51906.scom-troubleshooting-disabled-sql-server-service-broker.aspx which seems to be the solution on our SCOM implementation!
After correcting this issue UR2 ran smoothly
I have an issue with our SCOM 2019 Roll Up. It always sits at the the window that say “Removing Backup files” and I have left it there for 24hrs and does nothing. the end of the log say “:Install Progress – (RollbackCleanup) Removing backup files”
That is it. Any idea how I can get this installed? Our SCOM execution files are on a separate drive on F:\ don’t know if that matters
Can you send me your log file to kevin.holman@live.com
Simply installed the UR2 with the .exe on my management servers as the previous poster did. We have 4 MS accross 2 sites. With 2 in each site. This error only shows on the Management servers in a remote site. The SQL server they are all pointing to has the opsdb as well as the opsdw.
Another person with the stuck “Removing backup files” using the KB4558752-AMD64-Server.exe (run from admin command prompt). Tried and failed on different SCOM servers. Servers were already at UR1. Even worse, the manual KB4558752-AMD64-Server.msp now gets stuck as well – Possibly the installation has “worked” as in the console under ADMINISTRATION | Operations management Products | Management Servers shows update rollup 2 patch, and re-running the MSP not gracefully detecting this, and freezing (maybe). I’m refusing to go live with this though as I can’t guarantee stability.
Also had issues with the SQL update process which failed – See below for another link with people having similar SQL UR2 upgrade issues as I have had too. Looks like it’s gone off to the dev team. I went back to the old TSQL script process for now. Looks like the new EXE patcher needs some patching 😉
https://docs.microsoft.com/en-us/answers/questions/69612/scom-2019-ur2-installation-issue.html?page=2&pageSize=10&sort=oldest
Hi Kevin – Spoke too soon – the manual TSQL script process also has some issues:
The data warehouse script:
\Program Files\Microsoft System Center\Operations Manager\Server\SQL Script for Update Rollups\UR_datawarehouse works,
The ops TSQL script:
\Program Files\Microsoft System Center\Operations Manager\Server\SQL Script for Update Rollups\update_rollup_mom_db fails with the following message:
Msg 2714, Level 16, State 5, Line 3934
There is already an object named ‘FK_MaintenanceModeStatus_BaseManagedEntity’ in the database.
Msg 1750, Level 16, State 1, Line 3934
Could not create constraint or index. See previous errors.
.. On the microsoft docs site (below), there’s talk about modifying the script by hand to remove the issue, and also talk of totally ignoring this (8000 line!) script – I’d like to know what the best way forward is..
https://docs.microsoft.com/en-us/answers/questions/105423/scom-2019-ur2-patch-update-sql-script-error-there.html
When we run the setup even with Elevated rights, the Setup hangs at “Removing Backup Files” ..
It stays there forever, Unless we cancel it.
The Server stop receiving alerts and the clients report error 20070 (the agent is not autorized to comunicate with the server).
And the agents are always: “update in progress”
I am trying to run the MSI installer on the management server for scom 2019 , and the setup is stuck , can anyone help
Kevin, have you seen any issues with the Web Console after upgrading to UR2. If the user is local administrator on the web console server, it works fine, but if they are not admins, the performanceview returns a 500 error. Stack trace: [HttpException (0x80004005): Error executing child request for /MonitoringView/ResultsViews/ViewTypePerformance.aspx]
Yes. I’ll post the fix in the morning.
I posted the fix to this issue in the Known Issues. Let me know if this fixes yours.
You’re awesome Kevin, that fixed it right up! Thanks for all the great content!
Good day, recently updated from SCOM 2019 UR1 => UR2 using one-click installer (KB4558752-AMD64-Server.exe), finished without errors, when finished – the server gone into reboot without notification (hence i suppose it’s a known issue sometimes). After that, applied WebConsole update (.msp) and Console update (.msp), gone for reboot one more time. Applied Reporting update after that to the DB server. So far so good. In the Console all agents gone to Pending states – so updated them as well, no problem. Still at the Console itself (Administration section) for whatever reason it shows Managements Server, Operations Console, Web Server – Update Rollup 1 Patch (10.19.10311.0) and NOT the Updated one. Databases and Reporting shows – Updated one UR2 – 10.19.10407.0. All Agents also shows UR2 – 10.19.10407.0. Console on the server itself (Help=>About) shows 10.19.10407.0. More to that Administration – Device Mgmt – Mgmt Server – shows UPDATED one 10.19.10407.0., Get-SCOMManagementServer shows UPDATED 10.19.10407.0. No errors or anything after installation at C:\Users\\Appdata\Local\SCOM\Logs or Windows Server logs (Operations Manager) -was checked right after UR install. Rebooted Mgmt and DB Server once more, flushed Health Service state as well – same thing, did not solve the issue. At Console Managements Server, Operations Console, Web Server shows Update Rollup 1 Patch (10.19.10311.0) whatsoever. Pretty weirded out by this so far. Anyone had such behaviour ?? Suspecting a bug or something ..could doubt the installation was smooth in the end (though no errors as mentioned) but Get-SCOMManagementServer shows UPDATED 10.19.10407.0 version, so… WEIRD. no any clue on the Net about such of an issue so far.. any help appreciated.
Hi Kevin, I am posting in the hope of helping somebody else with the same issue.
I had an issue in a customer environment (SCOM 2019 ur2 with SQL Server 2019 UR8)
The customer was not able to import a custom sealed management pack (that was imported fine in every other combination of SCOM/SQL Server)
The error returned was a
mpinfra_p_managementpackinstall failed with exception Conversion failed when converting from a character string to uniqueidentifier
After some initial troubleshooting
With a SQL Profiler we were able to identify the problem that was a custom Class with a property like this one
The issue was that the Stored Procedure was trying to convert the DefaultValue from string to guid, in order to import the MP correctly I had to remove that default value and recreate it with an Override.
Can you show the XML that was bad?
I am curious to the details as well. I see Kevin is asking for additional information for the XML portion. You mentioned a SQL profiler. Any tips?
This issue is also causing the SQL Dashboard view to fail in SCOM 2019 via the Web Console. For some reason the rich-client console simply leaves the SQL Dashboard – Azure SQL section and renders it in the console. But the Web View of the same SQL Status Dashboard is broken. It reference the Azure SQL management pack. When we try to import the Azure SQL management pack we see the same error.
I am having this exact same issue. SCOM 2019 UR2. SQL Server 2019 UR9. Trying to import the Azure SQL DB Management Pack v7.0.26.0 and it fails with that exact error.
https://www.microsoft.com/en-us/download/details.aspx?id=38829
****************************************
Error While Trying to Import
****************************************
ImportManagementPackWithResources
uuid:ce4cd322-d56f-47e0-8530-3f8bc32d080b;id=8
Database error. MPInfra_p_ManagementPackInstall failed with exception: [MP ID: 19f0a584-8bde-2a36-e607-05060e04959a][MP Version: 7.0.26.0][MP PKT: 31bf3856ad364e35] Database error. MPInfra_p_ManagementPackInstall failed with exception: Conversion failed when converting from a character string to uniqueidentifier.
: Database error. MPInfra_p_ManagementPackInstall failed with exception: [MP ID: 19f0a584-8bde-2a36-e607-05060e04959a][MP Version: 7.0.26.0][MP PKT: 31bf3856ad364e35] Database error. MPInfra_p_ManagementPackInstall failed with exception: Conversion failed when converting from a character string to uniqueidentifier.
An exception was thrown while processing TryImportManagementPackWithResources for session ID uuid:ce4cd322-d56f-47e0-8530-3f8bc32d080b;id=8.
Exception message: Database error. MPInfra_p_ManagementPackInstall failed with exception:
[MP ID: 19f0a584-8bde-2a36-e607-05060e04959a][MP Version: 7.0.26.0][MP PKT: 31bf3856ad364e35] Database error. MPInfra_p_ManagementPackInstall failed with exception:
Conversion failed when converting from a character string to uniqueidentifier.
Full Exception: : Database error. MPInfra_p_ManagementPackInstall failed with exception:
[MP ID: 19f0a584-8bde-2a36-e607-05060e04959a][MP Version: 7.0.26.0][MP PKT: 31bf3856ad364e35] Database error. MPInfra_p_ManagementPackInstall failed with exception:
Conversion failed when converting from a character string to uniqueidentifier.
Same issue, is there a solution ?
Set your OperationsManager database to SQL 2017 compatibility. Then import the management pack.
Thanks Kevin, I had been wrestling with this for a few hrs. Lowering the db to comp level 140 fixed that import error for me.
Hi Kevin,
we have a brand new installation of SCOM 2019 the update on the server was smooth, the problem is with the additional consoles deployed on managing pc (windows 10 ver 2004), applying the KB takes a long time and at the end the windows closes without any error message, if I try to take a verbose log of the process with msiexec /upgrade ecc… i get an error
MSI (s) (8C:38) [12:32:41:303]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSID802.tmp, Entrypoint: StartNamedServices
CAStartServices: CAStartServices was passed . HealthService System Center Management APM
GetWindowsServiceAccount: OpenService failed. Error Code: 0x80070424.
CAStartServices: We could not get service start type, assuming the service is not installed, skipping service start. Error Code: 0x80070424.
GetWindowsServiceAccount: OpenService failed. Error Code: 0x80070424.
CAStartServices: We could not get service start type, assuming the service is not installed, skipping service start. Error Code: 0x80070424.
and it rolls back to the previous version.
The Console was installed with the original setup media for SCOM2019 with no issues, searched with google for possible causes but looks like I’m the only one with this problem….. sigh
thx for all that You share.
Is this a stand alone server with a console only? What version is the current console? What MSP file are you running? Are you calling the file from an elevated command prompt?
Hi,
its a win 10 machine with only the console installed, the console version is 10.19.10505 (retail)
the file I’m running is KB4558752-AMD64-Console.msp and yes from an elevated prompt.
cheers
Hi Kevin,
Quick info about the HTTP 500 issue with web console : I was facing a very similar issue and fixed it using your workaround, however the context with a bit different so it might help others.
In my case, the non-admin users were not scoped (native Operator/Advanced Operator roles) and were using Windows authentication; and the http 500 error only happened when selecting a perf counter in a perf view, not anywhere else as far as I could tell.
Hi Kevin, can I query for the UR version of the agents on the fly? I have your admin pack installed but it’s not immediate.
You can, but it would be one by one via a task. Quite painful. What are you trying to accomplish?
When the agent is patched, it should be restarted. When the agent restarts, my discovery runs within 4 minutes, and updates the SCOM Agents version view in the console within 2 minutes after discovery data is submitted. That’s pretty fast.
Or, you could simply override the discovery of the SCOM management agent properties, to run more frequently when you are doing upgrade work?
Pingback:Coffee break: SCOM Update Rollup 2 | SquaredUp
Hi Kevin,
We are having issues on SCOM 2019 installation.
We rebuilt the database from scratch and we are try to reinstall the SCOM management servers. During installation we encountered the error “the specified database is already in use”. Please guide what steps are missing.
You cannot rebuild the DB from scratch. That is not supported. SETUP creates the databases and assigns permissions. You should follow the product documentation or my quickstart guides.
https://docs.microsoft.com/en-us/system-center/scom/deploy-overview?view=sc-om-2019
Thanks Kevin for the quick response.