KB Article for OpsMgr
List of New Features
Download Update Rollup from the Catalog
Download the NEW Simplified Management Server Update EXE
Download Updated UNIX/Linux Management Packs
Recommended hotfix page
New Features:
- Updates to the new change tracking feature
- Additional view options in web console widgets
- Resolved issues with orphaned alerts
- Performance Improvement in load time for Windows computer view
- Performance Improvement in load time while changing user role privileges
- Performance Improvement in SDK service queries
- Performance Improvement in grooming of maintenance mode staging table
- UNIX/Linux – Reliability and performance improvement in Xplat agent
- UNIX/Linux – Support for RHEL 6
- UNIX/Linux – Disabled SSL renegotiation for Linux agent
- UNIX/Linux – TLS 1.2 support for Solaris 10 SPARC
- UNIX/Linux – Dynamic changes in log-level settings without agent restart
Key fixes:
- See the KB article. There are too many fixes to list here.
NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2019 and never applied an update rollup – you can go straight to the latest one available.
Let’s get started
From reading the KB article – the order of operations is:
- Install the update rollup package on the following server infrastructure:
- Management Servers
- Web Console Servers
- Gateway Servers
- Operations Console Servers
- Reporting Server
- Apply Agent Updates
- Update Unix/Linux MP’s and Agents
Management Servers
Updates in SCOM 2019 have CHANGED. There is a new process for updating management servers that differs from previous versions of SCOM. Download the single file management server update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates.
- Download: HERE
It doesn’t matter which management server I start with. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.
I have 2 management servers My first management server holds 3 roles, and each must be patched: Management Server, Web Console, and Console.
The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.
Notice the new EXE file, and a MSP file exist for the Server update. The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching. I will ONLY demonstrate and recommend the EXE file for the Management Server role update.
Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.
- ***Note: One of the changes in SCOM 2019 Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL. The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases. You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.
My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.
I will start with KB4594078-AMD64-Server.exe
This is a self-extracting executable, that kicks off a simple update tool. Accept the license terms, and click “Install”
This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.
If you have an issue – you can review the setup logs:
- Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs
- SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
- MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log
Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update: KB4594078-AMD64-WebConsole.msp
Lastly – install the Console Update (make sure your console is closed): KB4594078-AMD64-Console.msp
You can reboot the server at this time if you were prompted to in order to complete the update. If you were not prompted to, you do not need to.
Additional Management Servers:
Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers. You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.
You can use the same EXE file and MSP files (where applicable) you used for the first management server. The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if needed.
Updating Gateways:
Open an elevated command prompt, and run the update: KB4594078-AMD64-Gateway.msp
The update launches a UI and quickly finishes.
Updating Reporting:
On your server that hosts the SCOM Reporting role, run the update: KB4594078-AMD64-Reporting.msp
Update Agents:
Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):
***NOTE: For this to work, you MUST run the server update from an elevated command prompt, and the user account running the update must be a Local Admin, and SCOM Admin. The Agents MUST have “Remotely Manageable” set to “Yes”.
You can approve these – which will result in a success or failure message once complete:
Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:
And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery. This has long been a pain point in SCOM:
https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/
Update UNIX/Linux MPs and Agents:
You can get the current Unix/Linux MP updates HERE.
Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor. Here is the FULL list:
In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:
This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.
Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits. If they are not present, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an MP Import to get them to show up.
After restarting my Microsoft Monitoring Agent service on my management server, I see the new files dropped with new timestamps:
Now you can deploy the Linux agent updates:
If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:
Update the remaining deployed consoles
This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.
Verifying the update
There are new views in the SCOM console to help with this and make this process MUCH easier. You do need to wait long enough for the discoveries to run in order for these to update the views.
Review:
Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.
Known Issues:
1. The Web Console views may not work (500 error) for scoped users who are not SCOM Admins. These same views work fine for SCOM Admins.
- This primarily affects Web Console roles installed using Network Authentication. This does not appear to impact Web Console servers installed using Windows Authentication.
- On the web console server – find the folder: \Program Files\Microsoft System Center\Operations Manager\WebConsole\MonitoringView\TempImages
- Change the security on this folder for Authenticated Users – and add “Write” capability:
2. Once you apply an update rollup, you cannot install, or re-install SCOM WEB CONSOLE, or SCOM REPORTING roles.
- You will see an error in the setup UI (when you supply a management server name) that states “Unable to connect to the Data Access service for this management server. Ensure the Data Access service is running and that the service, the management group, and setup are all the same version.”
- In the setup log, located at C:\Users\<username>\AppData\Local\SCOM\LOGS\OpsMgrSetupWizard.log – the last line will appear similar to: Error:The management server is a different version than the current setup build. Please use a different management server or the correct version of setup. Server Version: 10.19.10505.0
- SCOM Reporting Setup looks for a very specific value in the SQL database, which has been updated by the Update Rollup and setup is now blocked.
- Apply the following workaround to install/reinstall SCOM WebConsole or Reporting role:
- QUERY the OPERATIONSMANAGER database, and record the VERSION number that is returned. You will need this value later. You need to change the PrincipalName to your SCOM Management server that you point the reporting install to.
-- 10.19.10050.0 - 2019 RTM -- 10.19.10311.0 - 2019 UR1 -- 10.19.10349.0 - 2019 UR1 with post UR1 Hotfix -- 10.19.10407.0 - 2019 UR2 -- 10.19.10505.0 – 2019 UR3 USE OperationsManager SELECT PrincipalName, Version FROM MTV_HealthService WHERE IsManagementServer = 1 AND PrincipalName = 'OMMS1.opsmgr.net'
UPDATE the VERSION entry in the OpsDB to match the RTM version number which is 10.19.10050.0 just for this management server.
UPDATE MTV_HealthService SET Version = '10.19.10050.0' -- 2019 RTM WHERE PrincipalName = 'OMMS1.opsmgr.net'
Install SCOM 2019 Reporting, and choose this same Management Server. Reporting install will work now.
REVERT the VERSION entry in the OpsDB to match the original value you recorded
UPDATE MTV_HealthService SET Version = '10.19.10505.0' -- 2019 UR3 WHERE PrincipalName = 'OMMS1.opsmgr.net'
3. When you run the KB4594078-AMD64-Server.exe server update, sometimes the tasks fail to update the SQL databases, or perform the Agent Pending Management task.
This can happen when your permissions are not set correctly for your RunAs accounts. There are 3 tasks that get deployed from the Management Pack included in the Update Rollup:
MP: Microsoft.SystemCenter.DBUpdateTask
- Database update task
- Datawarehouse update task
- Agent pending management task
The “Database update task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Datawarehouse update task” runs as whatever action account is associated with the RunAs Profile: “Data Warehouse Account”. By default the RunAs account “Data Warehouse Action Account” is associated with this profile, so the task will execute as the credential configured in the “Data Warehouse Action Account” RunAs account, which should be the Data Warehouse Write Action account you specified when you installed SCOM. This credential has a very high privilege to the OperationsManagerDW database (db_owner) so it can modify anything necessary. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
The “Agent pending management task” runs as whatever action account is associated with the RunAs Profile: “Operational Database Account”. By default there are no RunAs accounts associated with this profile, so any workflow that attempts to use this RunAs profile while it is unassociated will execute as the Management Server Action Account. Since the MSAA has a high level of privilege to the OperationsManager database, this will be successful. It will fail if someone has restricted the rights of this account on the management server or the SQL database/instance.
If any of these fail, it is likely that someone has modified the default permissions for your action accounts to the SQL databases, or someone has incorrectly modified the default RunAs profile associations. Please review your permissions against: SCOM 2019 Security Account Matrix – Kevin Holman’s Blog
You can always just run the SQL script update tasks manually, by using the method shown in previous versions of SCOM as seen here: UR10 for SCOM 2016 – Step by Step – Kevin Holman’s Blog
Done!
As always, thanks for the quick release of the update document.
The UR3 KB does not talk about the security vulnerability.
How can we confirm that this CVE is also fixed in UR3?
https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1728
Not sure why they didnt include that in the KB article. However, I tested it and it was included.
Hi Kevin,
Many thanks for the confirmation.
Another important feedback. The UR3 upgrade with exe file went well for the first management server, but my other managment servers were restared without giving any imtimation.
So if you dont have a reboot approval, you have to be careful.
Log Name: Application
Source: MsiInstaller
Event ID: 1005
Task Category: None
Level: Information
Keywords: Classic
Description:
The Windows Installer initiated a system restart to complete or continue the configuration of ‘System Center Operations Manager Server’.
I saw it in the KB on the 31st, so it was there, but it looks like the article has been pulled for one reason or another.
Saw it here: https://support.microsoft.com/topic/a7b9aa06-7d72-dc5a-e303-1899fafe9931
I previously had the issues with some Management server continuing to try to deploy warehouse components. Still occurs after this update although it says it was addressed in the patch. I tried both the simple installer then manually installing the UR on the affected servers. Weirdly this only occurs on management servers at a remote site but not on the local. Thanks:
Failed to deploy Data Warehouse component. The operation will be retried.
Exception ‘DeploymentException’: Failed to perform Data Warehouse component deployment operation: Install; Component: Script, Id: ‘4173391e-ce71-1369-c358-17030bb09b41’, Management Pack Version-dependent Id: ‘73695800-a0e6-5cc8-6a66-9f8bbe722d7f’; Target: Database, Server name: ‘ServerA’, Database name: ‘OperationsManagerDW’. Sql execution failed. Error 777971002, Level 16, State 1, Procedure DeploymentOperationValidate, Line 278, Message: Sql execution failed. Error 777971104, Level 16, State 1, Procedure DeploymentOperationValidate, Line 170, Message: Deployment operation validation failed. Component alrady installed. Operation: ‘Install/Upgrade’; Component type: ‘Script’; Component id: ‘4173391E-CE71-1369-C358-17030BB09B41’; Management pack version: ‘73695800-A0E6-5CC8-6A66-9F8BBE722D7F’; Target: ‘Server name: ServerA; Database name: OperationsManagerDW; Dataset ID: [null]’
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Component
Instance name: 8a9be2d2-02de-4f28-85a8-2ebf425c9dbb
Instance ID: {40D8D96F-675F-D7D0-019D-6E0E8C6CF9E2}
Ran ur 3 manuelly i have 20 agents still not updated. Should i run it again?
I’d run a repair on those then investigate the logs if it failed.
Hi Kevin, first of all many thanks for your solutions and recommendations, usually use them for my SCOM.
UR3 done successfully except agents, some of them were not updated from console and still have old versions 10.19.10014.0 and 8.0.13053.0
Do I need to update them manually (one by one) or better to find issue and update them using console?
Thanks in advance.
You can try to do a “repair” using the console. Then go investigate the ones that failed and see what is wrong/failing, then address it. If the number is manageable then yes, I’d expect you would fix those manually.
A UR update generally will not work on old agents, you need to upgrade them first…. I’d expect those to fail.
Hi Kevin,
do you have any news about the Ubuntu 2004 LTS support?
It has been a full year since the release, but there it’s still not supported by SCOM.
I recommend you open a request at aka.ms/SCOM (or a support case) for this. I am not sure why we have not added this support to SCOM 2019.
Hello!
First, many thanks for a great UR3 guide…
After we upgrading Scom agents to UR3 on our DC:s, they been consuming almost all of the CPU rersources. 99-100%. It is not only one, theres all of the DC:s…
IT is the “System Center Management Service Process” —> MonitoringHost who takes it all.
When we rolling back agents to UR1 there back to normal CPU consuming again. No problem at all…
We scanned the processes on a DC with Ur3 agent installed and there seems to be alot of Cscript activity going on.
Anyone else who has facing this issues with the UR3 agent?
That’s very concerning. Can you open a support case on that?
What Active Directory management pack are you using – the “new/current” on or the old one?
Yes thats is very concerning.
The Active Directory Management Pack we use is version 10.0.2.2, i guess thats the newest one?
Is there no changes for the AV exclusions with UR3?, same exclusions as before?
Well we have something to work with here and we will surely open a case on this, if we not find any solution. If anyone else also have this problem please confirm…
We found the root issue now. It is the “Security Monitoring MP” in combination with “SCOM UR 3 Agent” who is the problem. Security Monitoring MP created floods in the eventlog on the DC:s and when we uninstalled the Security MP everything went back to normal and the UR3 agent on the DC working just fine. It was Security Monitoring MP 1.0.7.1 had installed… Maybe Nathan Gau can knows or can find out what can be wrong here…
Thank you for sharing the information.
Your welcome. Mates in the same kind of boat surely row best togheter :=)
I am attempting to reproduce this
Thanks Nathan! Hope you can find something.
We made a test to install the Security Monitoring MP 1.0.7.x again. This time completely (vanilla), no overrides. The CPU consuming directly rised again.
When i looked trough the Eventlog under Operations Manager there is a lot of errors associated to Security Monitoring MP rules. For an example:
GPODeletionRule
GPOCreation
DCOUModify
There is also warnings that rules/monitors failed and got unloaded. 1 of them reached the failure limit that prevents automatic reload.
I dont really know but the feeling i have is that something is stucked in loop because the errors and warnings come intermittently. Considering reload and intermittently errors.
Hope this can give something…
I am seeing a LOT of handle count leaks and high CPU on all agents (especially DC’s) when the Security Monitoring MP is imported. I’d recommend removing this MP until Nathan has time to review what’s going on. It is a VERY intense MP with regard to the security event log, which is very difficult to monitor because of the number of events per second typical on DC’s.
Thank you Kevin for confirming that, and yes we will remove this meanwhile.
Hi,
I reported the handle count issue to the PG and MSFT has released a Hotfix for that:
https://support.microsoft.com/en-us/topic/system-center-operations-manager-2019-hotfix-oct-2021-f5aa7ec5-85c2-4886-b13b-288fd9900737
Cheers,
Patrick
Hi Guys, Hi Kevin,
I am currently installing the UR3 in two different, independant environments. Unfortunately I am facing in both of them an issue, which should have been resolved with UR2:
– Setup stalling at the “Removing backup files” stage.
I am currently running UR2 in both environments and the UR2 setup went nice and smooth, but the UR2 stales at this phase, waiting for 45 Min. now, pretty sure it won’t continue.
Anyone else experiencing this? Any advices? Thanks in advance!
Ok, seems to be false alaram. It was just the update, taking very, very long (about 55 Min.) to get from “Removing Backup Files” to the next stage!
Weird. Sounds like their “fix” was to just put in a timeout that eventually lets you continue.
Thanks for this excelent post as always Kevin!
Grettings from Colombia!!
The catalog seems jacked up and I can’t download the UR3 suite of installers. Is there another download location?
I just tested it. Catalog working just fine here.
Use ms edge instead of chrome if you are trying that… chrome didn’t work for me.
Hi
Thanks for the post, always a joy to read and follow.
A comment to STOYAN CHALAKOV above. Removing Backup Files took at least 1h before it continued.
The next step, updating database, I received error 1603 during DataWarehouseUpdateTask and the process stopped.
Found an old post stating that that error is nothing to be concerned of, just continue to install the MPs manually, which I did. The databases has the right version number in the consol. Anyone else experience this? Any comment?
Had a fail on the 2nd MS on the “Database Configuration” stage when using the server.exe. Rebooted it and still did the same thing. Tried a 3rd MS and it was good. Went back to the 2nd one and used the old .msp method. Everything looks ok but looks like this issue is still around:
https://docs.microsoft.com/en-us/answers/questions/69612/scom-2019-ur2-installation-issue.html
As the 1st MS completed the Database Configuration and MP sections cleanly, I assume that it doesn’t matter it failed on the 2nd one anyway, but ran the .msp just incase (even though the “Management Server” section completed OK from the initial attempts using the server.exe.
Has anyone else had this issue and do I need to do anything else?! Thanks
I have similar problem. Both tasks “AgentPendingMgmtTask” and “DatawarehouseUpdateTask” are failed by timeout but they last less than 15 seconds on management servers. Looks like they are finished faster than installer receives status.
These tasks Use RunAs accounts/profiles to do their work. If they fail, you can run the SQL script manually. But you really need to review the rights and compare to my SCOM 2019 security matrix: https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/
The most likely cause is messed up permissions. I added a new bullet #3 to the known issues. Check it out.
After updating to UR3 we noticed that Alert Monitor “Processing Backlogged Events Taking a Long Time” would go in to a warning state for a few servers, and a critical state for our domain controllers. We never had this issue before with blacklogged events. We would see the alert, but it would automatically correct easily. It’s been in a critical state on our DCs for several hours today. We’ve been troubleshooting, looking at the source of the logs, trying to find out why this is suddenly an issue. We even flushed the Health Service State and Cache and rebooted our SCOM servers (we have 2). We don’t monitor workstations, and the total of our monitored clients is roughly 300-400. We believe we have narrowed it down to UR3, as we just updated. Is anyone else having this issue after updating to UR3?
Hi David!
In earlier comments in this thread, i have been reporting problems with DC:s and very high CPU Consuming with the UR3 agent. Maybe you already have read it. That issue seems to be a problem with the UR3 agent and the Security Monitoring MP. We have uninstalled the Security Monitoring MP and the CPU Consuming went back to more normal consuming…
BUT!
We have also seen the Processing Backlogged Events Taking a Long Time. We get that alert often for one of ur Management Server. Before UR3 we never had this. So my guess is that Security Monitoring is not the only problem with UR3. The feeling that i have is that UR3 consuming more resources, both cpu and memory.
So i would not be suprised if there soon will be a hotfix for UR3. Hope they working on it….
Martin,
Thank you! I must have missed it. We do use the security monitoring MP. It’s one of our most valuable MPs. Our cpu utilization on the DCs has come down after overriding the monitor. We left it on for all of our other servers. I’ll have to turn those overrides off to see the results of the next patch.
Hi Martin
How are you doing?
Did you notice any problems with the agent on servers with SQL Server in your environment? Or only in DC’s?
Hi Rodrigo!
Thanks, I’m feeling very well and i wish the same for you.
No generel SQL Server problems what I’ve seen for so long. But the SQL servers dedicated for SCOM are definitely using more resources than before. This can possibly be a consequence of the other problems, feel insecure about this.
Except the DC´s, one of our Management Server has problem with “Processing Backlogged Events Taking a Long Time” and are also consuming more resources then before. it feels like it is increasing gradually. Before UR3 we did not have this.
Hi Martin
Thank you for your feedback.
Thank you for this guide!
The link to the new linux agents is broken, it was working yesterday 19/05/2021
Thanks a lot ..and here is my little issue. We have a couple 32-bit Windows 10 with SCOM agent . Where could I get UR3 update for them ? There is only 64-bit update for MMA presented on official site.
SCOM 2019 only supports 64 bit agents: https://docs.microsoft.com/en-us/system-center/scom/system-requirements?view=sc-om-2019#microsoft-monitoring-agent-operating-system
UR3 is irrelevant… as we do not support x86 agents for SCOM 2019.
Thanks a lot !
Shame on me, I shoud read docs carefully.
Update KB4594078 was revoked? https://www.catalog.update.microsoft.com/search.aspx?q=4594078 doesn’t return result.
Not revoked – just a temporary issue with the catalog for UR3
We have some servers with SCOM agents running version 10.20.18040.0 are the also affected by this vulnerability?
https://nathangau.wordpress.com/2021/06/03/update-on-security-monitoring-and-ur3/
Hello,
Thank you very much for this clear article ! I’m also experimenting issues with UR3 I don’t use the Security Monitoring management pack !
My Agents are multihomed to Azure and in Azure we are doing AD Assessment and Security monitoring and some other check that need event logs to be read 🙂 In fine, Agents are consumming Processor, they are forced to restart every 30-40 minutes by the Monitoring host handle count monitor, Event log are wrapped (you can see event 26013 in Operations Manager Event log) and agent queue is full and filling up… You loose Data in the performance collection. I’ve an opened call at Microsoft related to taht point and they confirmed this issue as a bug. They are working on it.
As a workaround, I’ve been asked to rollback agent to UR2.
So if you are using SCOM agent to send Data to azure, forget UR3 for the moment.
Thanks for your information Tristan! So the rollback to UR2 is the only workaround for the moment? Did they announce a (aproxdate) for the real fix?
Manually Installing UR3 SQL Script on the Datawarehouse results in “ANSI_PADDING” errors when executed. Here are a few lines from the error output…
Msg 1934, Level 16, State 1, Procedure trDBMon_TrackServerSecurityEvents2, Line 22 [Batch Start Line 7897]
INSERT failed because the following SET options have incorrect settings: ‘ANSI_PADDING’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
Msg 1934, Level 16, State 1, Procedure trDBMon_TrackServerSecurityEvents2, Line 22 [Batch Start Line 8142]
INSERT failed because the following SET options have incorrect settings: ‘ANSI_PADDING’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
The module ‘ChangeTrackingMaintenance’ depends on the missing object ‘dbo.ChangeTrackingAgentMaintenance’. The module will still be created; however, it cannot run successfully until the object exists.
The module ‘ChangeTrackingMaintenance’ depends on the missing object ‘dbo.ChangeTrackingResetMonitorMaintenance’. The module will still be created; however, it cannot run successfully until the object exists.
Msg 1934, Level 16, State 1, Procedure trDBMon_TrackServerSecurityEvents2, Line 22 [Batch Start Line 8161]
INSERT failed because the following SET options have incorrect settings: ‘ANSI_PADDING’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
Msg 1934, Level 16, State 1, Procedure trDBMon_TrackServerSecurityEvents2, Line 22 [Batch Start Line 8238]
INSERT failed because the following SET options have incorrect settings: ‘ANSI_PADDING’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
Msg 1934, Level 16, State 1, Procedure trDBMon_TrackServerSecurityEvents2, Line 22 [Batch Start Line 8318]
INSERT failed because the following SET options have incorrect settings: ‘ANSI_PADDING’. Verify that SET options are correct for use with indexed views and/or indexes on computed columns and/or filtered indexes and/or query notifications and/or XML data type methods and/or spatial index operations.
ANSI_PADDING is usually a client configuration issue. What version of SQL management studio? Did you try from a different SQL management studio?
Kevin, my install keeps failing at Install Datawarehouse update task. I checked the security matrix, and even gave my service accounts sysadmin rights in SQL. I still keep failing. I tried to update using the .msp files, but even that failed. Could this be because I switched to using gMSA accounts?
Interesting. I have still not tested gMSA yet. Quite possible.
Can you run the update manually using SQL?
Hello Kevin, I have a problem with the new Linux Agent 1.6.8-0 can’t discover Jboss EAP 6.4.0. Y get the error:
2021-06-11T18:14:11,560Z Error [scx.core.common.pal.system.appserver.appserverinstance:877:2813:139914970294400] JBossAppServerInstance::UpdateJBoss5Ports() – {JEE Application Path} – Could not find file: {JEE Application Path}/conf/bindingservice.beans/META-INF/bindings-jboss-beans.xml
How I can restore the Agent version 1.6.3-793, where I can download that version?
does 2019 UR3 support Server 2022?
Not yet. It will be posted to the supported configuration pages when/if it does. In my testing, it seems to work just fine.
Hi Kevin,
since I updated our SCOM 2019 installation with UR3 I get a lot of Alerts in regards to our SPN. Like: The System Center Data Access service failed to register an SPN. A domain admin needs to add MSOMSdkSvc/SERVERNAME. When trying to set the SPNs I get the message “Duplicate SPN found, aborting operation!”
any ideas?
thx and best regards
Geri
Are your SPN’s set correctly?
https://kevinholman.com/2011/08/08/opsmgr-2012-what-should-the-spns-look-like/
If so – ignore the error and disable the rule. If not – fix it.
Hi,
I can find the SPNs for all 5 of my Management Servers. The SPN for the DataAccessService is missing. When I try to create it with the “Setspn -s MSOMSdkSvc/ServerName Doamin\SDKAccount” it geht the “Duplicate SPN found, aborting the operation!”.
Funny thing is, this message wasn´t present before I updated to UR3.
And ideas :-)?
thx and best regards
Geri
This is, how a SPN Query to one of the 5 Servers looks like:
C:\>setspn -l SCOMServer
Registered ServicePrincipalNames for CN=XXX,OU=XXX,OU=XXX,OU=
XXX,OU=XXX,DC=XXX,DC=XXX,DC=XXX:
MSOMSdkSvc/SCOMSERVERxxx.xxx.xxx
MSOMSdkSvc/SCOMSERVER
MSOMHSvc/SCOMSERVERxxx.xxx.xxx
MSOMHSvc/SCOMSERVER
WSMAN/SCOMSERVER
WSMAN/SCOMSERVERxxx.xxx.xxx
TERMSRV/SCOMSERVERxxx.xxx.xxx
TERMSRV/SCOMSERVER
RestrictedKrbHost/SCOMSERVER
HOST/SCOMSERVER
RestrictedKrbHost/SCOMSERVERxxx.xxx.xxx
HOST/SCOMSERVERxxx.xxx.xxx
Follow the article. Your SPN’s are wrong. The SDK SPN’s should not be set on the SCOMServer computer account. UR3 actually “fixed” this to check and alert correctly. It’s been broken since SCOM 2007 shipped and I have been asking for it to get fixed for 13 years. 🙂
Hi Kevin,
After updating the clients to ur3 i’m seeing on a lot of systems. Event id 1013: Product: Microsoft Monitoring Agent — Upgrade is not supported for the currently installed version. Please refer to the Setup and Deployment guide for more information on supported upgrade scenarios.
6 times and about a day later same process starts again.
What could be the cause of this?
I have not seen this. Sounds like you have SCCM or some other software distribution triggering. SCOM doesn’t do this natively. Are you using Azure MMA mixed in the environment?
yes, using Azure MMA mixed in the environment. Small number of machines are dumping data to log analytics. All machines have workspace configured for ATP.
Found it, on some machines sccm policy was enabled to push the client for endpoint protection.
Thx for pushing me in the right direction.
Thx Kevin for clarifying. I already read the article but wanted to make sure that I really got it right :-). Thx for your support and have great day!!
best regards
Geri
Hi Kevin,
I have updated successfully including the MPs however still failing to update the RHEL6 agents. Its still giving “Value cannot be null. Parameter name: lhs”
Have you tried removing the agent completely and re-installing it? This agent might not be upgrade compatible. If that doesn’t work I’d recommend opening a support case.
Thanks Kevin,
Removing the agent, provides me with the below error when trying to discover it again.
Unexpected DiscoveryResult.ErrorData type. Please file bug report.
ErrorData: System.ArgumentNullException
Value cannot be null.
Parameter name: lhs
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions)
at Microsoft.SystemCenter.CrossPlatform.ClientActions.DefaultDiscovery.InvokeWorkflow(IManagedObject managementActionPoint, DiscoveryTargetEndpoint criteria, IInstallableAgents installableAgents)
Following the above, i have removed the agent -> installed the agent manually -> discovery hanged in Managing but was still successfully discovered and added. Although healthy, pressing on “upgrade agent” still pops up the error Parameter incorrect.
As an update, issue was due to old omsagent agent files still present in the linux agents directory
Hello Kevin! Do you know if a Hotfix or UR4 will come in the near future. It feels a little bit frustrating to be stuck since March/April with the UR3 issues. I thought a quick hotfix would be released for this given the issues.
Update! After contact with MS, there is now a Hotfix available “System Center Operations Manager 2019 Hotfix – Oct 2021” KB5005527.
https://support.microsoft.com/en-us/topic/system-center-operations-manager-2019-hotfix-oct-2021-f5aa7ec5-85c2-4886-b13b-288fd9900737
Just now starting to investigate putting UR3 on my test SCOM servers, and as usual Kevin Holman’s article and all the various comments and replies I have read through are invaluable. My question is whether this hotfix is only necessary if running the Security Monitoring MP/connections to Azure?
As per article, the hotfix fixed an issue causing High Handle Count and CPU Utilization due to additional processing needed to resolve the formatted strings, in case of high volume of monitored events. Operations Manager Health Service will no longer resolve formatted strings which avoids additional processing.
Prior applying the hotfix i could notice a large number of agents peaking the CPU handle count threshold.
Did anyone have similar issues with UR2? I have purposefully not upgraded to UR3 because of the high CPU usage issues some have had. But, after some new security event logging was enabled recently, that has caused an enormous amount of events in the Security Event Log, I am seeing 100% CPU usage on quite a few servers. If I stop the MMA service, the CPU usage goes down. I am wondering if the issue they fixed in the hotfix for UR3 would help? I know it’s not for UR2, but if upgrading to UR3 and then applying the hotfix would resolve the issue, I would try it.
The issue is resolved in SCOM 2019 UR3 + Post UR3 hotfix KB5005527
Hi Kevin,
Thank you for great article as always
I am much curious to know about the Unix/linux environment like Cluster, Java, Tomcat, weblogic…etc. any update till scom 2019 UR3. Have you wrote any article on these how to monitor. Thanks in advance.
Afternoon,
My environment has been at 2019 UR3 for about 6 weeks now. A few days ago, we were trying to modify Subscribers and Subscriptions, when this error began showing up when attempting to Save:
“Saving notification subscriber… Failed to save the Notification Subscriber. The operation for the recipient could not be completed. See inner exception for details.” When I select the Error Details, nothing pops up..
I remember reading this was an issue in 2019 UR2 and there was a fix of reinstalling UR2.
https://docs.microsoft.com/en-us/answers/questions/648037/cannot-modify-create-save-notification-subscribers.html
Have you heard of this issue in UR3?
Can I reinstall UR3 without harm?
Run the sql update scripts again, manually? I used the new .exe to apply the UR3 update to both mgmt servers, which I understand also does the SQL part?
Assistance and time with this are immensely appreciated.
Tony
You can always re-apply the Update Rollup. If some step is not required, it will be skipped.
I am doing a brand new SCOM 2019 installation and Im pretty new to SCOM. What exactly do I do with agent update – i.e. KB4594078-AMD64-Agent.msp? Im guessing I dont run it on the Management Server, but I dont have any agents deployed yet so how do I make sure any new agents get the update?
Thanks
David Z
When you apply a UR to SCOM, any agents pushed after that UR will get it automagically.
Hi,
it´s stated that orphaned alerts are fixed in this UR, (also in this Doc: https://docs.microsoft.com/en-us/system-center/scom/manage-alert-created-by-monitor?view=sc-om-2019#closure-of-orphan-alerts)
However: We are facing many alerts which aren´t closed, even though the monitor is in a healthy state. This applies mostly to alerts, which are generated by “Database Status” Monitor from the SQL MP.
Anyone ran already into this issue ?
Thanks
Kamil
Hi Kevin,
Just a quick question, Do you have any information, when the UR4 can be released? I saw, that the SCOM 2022 has been released, but currently I’m more interested in UR4 currently. I searched everywhere, but couldn’t find any information.
Thank you very much,
Gergo
I am working on upgrading an environment to 2019 UR3 in preparation for SCOM2022. It seems that the Version number doesn’t not update on agents and Consoles? In the Operations Manager Products section under Administration, the consoles show the update installed but the Version number stays the same 10.19.10505.0.
In the control panel of agent servers after the update, the version number doesn’t update either. In your SCOM Administration MP, after 15 to 30 minutes it does update there.
Thanks in advance.