Menu Close

UR6 for SCOM 2016 – Step by Step


image

 

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/4459897/update-rollup-6-for-system-center-2016-operations-manager

Download catalog site:  http://www.catalog.update.microsoft.com/Search.aspx?q=4459897

Updated UNIX/Linux Management Packs:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page:  https://kevinholman.com/2009/01/27/which-scom-hotfixes-should-i-apply/

 

NOTE:  I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available.

 

Key fixes:

  • Application pool crashes and SharePoint application crashes because of APM agent and the profiler are now fixed. These occurred because the profiler was always on. This issue is fixed by disabling the profiler by default and enabling it only when APM is configured.  Note: If you’re using APM, restart IIS after you deploy the patches.
  • Fixed: Scheduled Maintenance Mode doesn’t work in an availability group that uses SQL Always-On configuration. In case of a failover to a secondary node, the Maintenance Mode Schedules that are created on the primary node don’t run. This has now been fixed. For details, see the “Enabling Schedule Maintenance Mode with SQL Always ON” section.
  • Updated fix: The Get-SCOMGroup cmdlet is slow to query large (more than 2,000) groups. A previous fix has been improved for faster querying of any number of groups.
  • Fixed: Using MI APIs causes MonitoringHost.exe to crash. This event is logged in the Application log, but no crash events are reported in the Operations Manager event log. After this fix, MonitoringHost doesn’t crash even if the useMIAPI registry key is enabled.
  • Fixed: SCX cannot connect with Linux machine if the KEX algorithms are configured as any of the following: ecdh-sha2-nistp256, diffie-hellman-group-exchange-sha256, or diffie-hellman-group14-sha256.
  • Improved logging to display error messages and suggest that you check performance counters when PdhExpandWildCardPath fails.
  • Fixed: Session Events tab in Application Diagnostics web console doesn’t display any data.
  • Fixed: Users could not edit the Maintenance Schedule in the SCOM console if all the objects were removed. The schedule can now be edited. However, it cannot be saved unless it contains at least one object.
  • OMS (now Azure Log Analytics) connection onboarding wizard is updated to communicate with the new APIs. This is because the OMS services were moved to the Azure Portal (link), and also because of the planned retirement of the OMS portal. This change will not affect any existing connection to OMS. However, for new connections or to reconfigure existing connections, the relevant management packs that are bundled together with this update rollup must be imported.
  • This update rollup creates a new “omi” user (system account).
  • XPlat support matrix changes:
    • SLES 10 is not supported.
    • CentOS 5 is not supported.
    • AIX 6.x is not supported.

 

Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management Servers
    • Audit Collection Server (ACS)
    • Web Console Servers
    • Gateway Servers
    • Operations Console Servers
    • Reporting Server
  2. Apply SQL scripts
  3. Manually import the management packs
  4. Apply Agent Updates
  5. Update Nano Agents
  6. Update Unix/Linux MP’s and Agents

 

 

WHOA Nelly.

Before we get started – there are some configuration changes necessary in SQL to support UR6.  UR6 has some fixes for Scheduled Maintenance, and these require some rights configuration FIRST – BEFORE applying UR6.  Otherwise you might run into some trouble.

Essentially, you need to:

  1. Open SQL Management studio
  2. Connect to the SQL instance that hosts your OperationsManager database
  3. Select your SQL login for the Management Server Action account, choose properties.
  4. Select “User Mappings”, and Add a user mapping to the MSDB database.
  5. In this user mapping – select db_owner, public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

image

 

  1. Now, open the properties for the SQL login for the SDK/DAS account.
  2. Select “User Mappings”, and Add a user mapping to the MSDB database (you may already have one).
  3. In this user mapping – select db_owner, public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

 

image

 

 

Note:   If you do not configure these rights, you might see failures in the SCOM logs after applying UR6 about not being able to synchronize scheduled maintenance jobs, or you might see issues with creating new Schedule Maintenance failing, locking up the console, or even creating large numbers of duplicate jobs.

 

 

 

 

1. Management Servers

image

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, and I always recommend a manual installation on management servers, I don’t recommend using Windows Update.  My first management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

 

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

 

My first server is a Management Server, Web Console server, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

 

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but you MIGHT see a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

 

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 6 Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute:

image

 

Next up – run the Web Console update:

image

 

This runs much faster.   A quick file spot check:

image

 

Lastly – install the Console Update (make sure your console is closed):

image

 

A quick file spot check:

image

 

Or help/about in the console:

image

 

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

 

Updating ACS (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

image

image

 

Note the above image states “Operations Manager 2012”.  This is a known issue.  Ignore it.

Updated files:

image

 

 

Updating Gateways:

image

Generally I can use Windows Update or manual installation.  I will proceed with manual:

image

 

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

 

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

 

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

 

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:

image

 

 

2. Apply the SQL Scripts

image

 

In the path on your management servers, where you installed/extracted the update, there is ONE SQL script file:

%SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment or clean install)

Next – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

 

image

 

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation. 

I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

 

3. Manually import the management packs

image

There are 36 management packs in this update!  Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language. 

This is the initial import list:

image

 

What NOT to import:

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operations Insights) and have your on premise SCOM environment connected to the cloud service.

DON’T import ALL the languages – ONLY ENU, or any other languages you might require.

The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (rare)

The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM.

So I remove what I don’t want or need – and I have this:

image

 

#Note: If the “Install” button is greyed out – this means you might already have one or more of these MP’s with the same version installed.  Find it by scrolling through each one, the console will tell you if you already have the same version.

 

 

4.  Update Agents

image

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.

You can approve these – which will result in a success or failure message once complete:

image

 

You normally can verify the PatchLevel by going into the console and opening the view at:  Monitoring > Operations Manager > Agent Details > Agents by Version

image

 

I *strongly* recommend you take a look at this community MP, which helps see the “REAL” agent number in the Administration –> Agent Managed view console:

https://kevinholman.com/2017/02/26/scom-agent-version-addendum-management-pack/

image

 

And my SCOM Management Group Management mp (updated for UR6), which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

 

5.  Update UNIX/Linux MPs and Agents

image

 

The UNIX/Linux MP’s and agents at the time of this article publishing have not changed since SCOM 2016UR3 was released.

You can get the current Unix/Linux MP updates here:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR6 is 7.6.1089.0  – and includes agents with version 1.6.2-342

 

 

Make sure you download the correct version for your SCOM deployment version:

image

 

 

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor:

image

 

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:

image

 

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

 

You should see the new files dropped with new timestamps:

image

 

Now you can deploy the Linux agent updates:

image

image

 

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

image

 

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://kevinholman.com/2016/11/11/monitoring-unix-linux-with-opsmgr-2016/

 

 

6.  Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

 

Review:

image

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

Known Issues:

1.  After applying UR6 – When a scoped Operator uses the console, it crashes with a specific error:    Incorrect syntax near the keyword ‘CREATE’

image

You will also see events in the OperationsManager event log on the management servers showing exceptions:

Source:        OpsMgr SDK Service
Event ID:      26319
Description:
An exception was thrown while processing GetRelatedManagedEntitiesByManagedEntityTypesAndCriteriaWithInstanceQueryOptions for session ID uuid:d35b8bee-4d1f-49ba-aeb3-0f4021d3eda0;id=221.
Exception message: The creator of this fault did not specify a Reason.
Full Exception: System.ServiceModel.FaultException`1[Microsoft.EnterpriseManagement.Common.UnknownDatabaseException]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Incorrect syntax near the keyword ‘CREATE’.).

Source:        DataAccessLayer
Event ID:      33333
Description:
Data Access Layer rejected retry on SqlError:
Request: MT_Select_Rel_ea99500d-8d52-fc52-b5a5-10dcd1e9d2bd — (ParentIds=457c209e-d14b-0721-be20-c3bf55447dab), (Depth=0), (UserRoleIds=e7cb00d7-9b7e-4657-8a0e-7479a32e1951), (UserScopeIds=S-1-5-21-3626055071-1639654894-2113106914-1112,S-1-5-21-3626055071-1639654894-2113106914-513,S-1-1-0,S-1-5-32-545,S-1-5-2,S-1-5-…), (OperationId=e8b526b8-2404-4b2a-ab56-db3d9c7ef6aa)
Class: 15
Number: 156
Message: Incorrect syntax near the keyword ‘CREATE’.

This is a bug, and will be fixed in UR7.  To work around this bug, you need to replace “Microsoft.EnterpriseManagement.DataAccessService.Core.dll” with a previous version, on all management servers.  This file is located in your Management Servers, in \Program Files\Microsoft System Center 2016\Operations Manager\Server.  The affected file version is 7.2.12066.0.  You need to rename this file to something like Microsoft.EnterpriseManagement.DataAccessService.Core.dll.UR6 then copy over the same file from UR5.  The UR5 version is 7.2.12016.0.  Stop the SDK/DAS service, rename the UR6 file, copy over the file from a UR5 system, and start the SDK/DAS service.

You can download UR5 from http://www.catalog.update.microsoft.com/Search.aspx?q=KB4090987

You can extract the KB4090987-AMD64-Server.msp using a unzip utility, and find the file “F_MEM.DAS.Core.dll.62894CB9_4320_40DB_B4E4_C0347FAB97B6”.  Copy this file and rename it to Microsoft.EnterpriseManagement.DataAccessService.Core.dll

 

2.  The ACS update shows “Operations Manager 2012” in the UI but is actually for SCOM 2016.

46 Comments

  1. Janez

    Kevin perhaps you know when the Application pool crashes and the Scheduled Maintenance Mode will be fixed in SCOM 1801/1807? With new release or only in SCOM 2019?

    • Kevin Holman

      I believe those will be fixed by SCOM 2019. In the SAC model, fixes arrived in the next version, which was every 6 months. The SAC model doesn’t have UR’s. Maintaining two branches can be a little confusing to customers, because different fixes will reach different branches at different times.

  2. andyinsdca

    The OM event log on all 5 of my management servers got corrupted after applying this. Had to kill HealthService.exe, clear the event log and restart healthservice. Other than that, it went fine.

    • Kevin Holman

      This is now interesting to me…. seeing how we reset the event log size in UR7. I wonder if we did this in UR6 and I just didn’t notice it.

  3. Gurunath Reddy

    Hi Kevin,

    Thanks for the detailed article. I have a strange issue here after applying the UR, Total process went without any error (Updated Server, web console, ACS, console and Agents as well.
    The problem here is that the Agent version is not updating in the console, it is still displaying the older version but under ‘Patch List’ it is correctly mentioned with UR 6 patch. Event in the registry of the Agent, it is showing older version. I tried manual installation also, still it is same.
    What is the way forward?, please help. Thank you

    Regards
    guru

  4. Gurunath Reddy

    Hi Kevin,

    Thanks for the detailed article. Everything went well for me with out errors but the agent version is still not updated. I tried manual update, still same issue. Even in the registry, it is not updated but in the ‘Agents by Version’, it shows UR6 Patch.

    what is the way forward, please help. Thank you

    Regards/guru

  5. Ahmed Essam

    Hi KEVIN,

    Thanks for the complete guide, I’ve an issues after applying UR6 on newly installed SCOM 2016 i noticed that there is alert (All Management Servers Pool Unavailable) following by EVENT ID 20022 (All Management Servers Resource Pool) is not heartbeating as well as Notifications Resource Pool

    AD Assignment Resource Pool

    This Server is newly installed

    Also i used your MP to get agent version its show me that the RU is 6 and the version is the latest but the version in managed computers still the old, i tried to repair – uninstall and reinstall same issue, i tried to discover new devices and install the agent on it i got the same issue

    appreciate your help

    Thanks in advance

  6. Sergey

    Hi Kevin,

    After Install SCOM UR6 we have problem with “My Workspace”.
    Users (Operator Role and Advanced Operator) can’t use saved State views and create worked new. Got Error:
    =====
    Date: 07.02.2019 17:37:02
    Application: Operations Manager
    Application Version: 7.2.12066.0
    Severity: Error
    Message:

    Incorrect syntax near the keyword ‘CREATE’.
    ==========
    In Operations Manager log:
    Level: Warning
    EventID: 33333
    Source: DataAccessLayer
    ====================
    Data Access Layer rejected retry on SqlError:
    Request: MTV_Select_ea99500d-8d52-fc52-b5a5-10dcd1e9d2bd — (UserRoleIds=e94743e9-b72a-41fe-858e-074cf81e2863,b1e29780-16dd-483d-84df-0ca8680caef6,2de7d87c-7ca7-4051-892f-1f7ffa43e8c2,9c886019-f9b8-47e…), (UserScopeIds=xxxSIDS_HERExxx…), (OperationId=e8b526b8-2404-4b2a-ab56-db3d9c7ef6aa), (NetbiosComputerName0=%%%)
    Class: 15
    Number: 156
    Message: Incorrect syntax near the keyword ‘CREATE’.
    ==========
    and
    Level: Error
    EventID: 26319
    Source: OpsMgr SDK Service
    ==========
    An exception was thrown while processing GetManagedEntitiesByManagedEntityTypesAndCriteriaWithInstanceQueryOptions for session ID uuid:8c32d870-9e7f-4e5d-817e-df2d361bb78e;id=17738.
    Exception message: The creator of this fault did not specify a Reason.
    Full Exception: System.ServiceModel.FaultException`1[Microsoft.EnterpriseManagement.Common.UnknownDatabaseException]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Incorrect syntax near the keyword ‘CREATE’.).
    ==========
    Uninstalling console patch not helping.
    Administrators not affected.

    Possible this is a bug.
    Any help will be appreciated.
    Thanks in advance!

    • Kevin Holman

      This is a known bug in UR6 for operators with scoped views. You need to back date one of the files from a UR5 install. I will edit the post to cover this in the known issues section

  7. Stoyan Chalakov

    Hi Kevin,
    we are really interested in the back dating the file you mentioned (UR5 Install), because we have face an issue with almost all scoped views – the console crashes with “Incorrect syntax near the keyword ‘CREATE’.” I found your comment yesterday and already opened a case to get the update, but I would be also interested in the possible workaround (or solution). Many thanks in advance!

  8. Pingback:SCOM 2016 UR6 console crashes with “Incorrect syntax near the keyword ‘CREATE’.” | POHN IT Blog

  9. JL

    Kevin, running into an issue where jobs are now failing with this error

    Alert: MSSQL 2016: A SQL job failed to complete successfully
    Source: SQLSERVERAGENT
    Path: SQLSERVER50.hq.first.int;MSSQLSERVER
    Last modified by: System
    Last modified time: 3/14/2019 11:30:16 PM Alert description: Event ID: 208. SQL Server Scheduled Job ‘BAE833BF-7318-E911-A2F4-005056923776’ (0x6EEC45C75FB68A40AC327423C9925DDF) – Status: Failed – Invoked on: 2019-03-14 23:30:15 – Message: The job failed. The Job was invoked by Schedule 21 (BAE833BF-7318-E911-A2F4-005056923776). The last step to run was step 1 (Maintenance Schedule Process Step).

    Alert view link: “?DisplayMode=Pivot&AlertID=%7b859aad82-8347-4f18-b56f-0504d39f162e%7d”
    Notification subscription ID generating this message: {7EE83EF8-D787-375F-786C-5797872B23FA}

    Any ideas?

    Thanks

  10. JL

    Just an update. I noticed that the agents never checked in for an update. I got to looking and UR 6 was attempted to be installed via SCCM and it failed (Could not stop services: Access Denied). I logged in to each MS and stopped the services. The update ran fine. All the agents started checking in for updated agent, but it didnt fix the maintenance schedule issues on a SQL AG. Still cant open them

  11. JL

    And another update on this for anyone else having issues. The SQL scripts runs when executing as a whole. It creates the new Stored Procs, but they do not get executed as I thought. According to this https://support.microsoft.com/en-us/help/4459897/update-rollup-6-for-system-center-2016-operations-manager#verify
    you need to manually run the stored proc (if you are impatient like me).

    After manually running the SP, the maintenance schedules were immediately available.

  12. Gerald

    I am getting the same error you mentioned here with scoped users in SCOM 2019, but it only happens with a scoped user in the Web console. They also get an error 500. I replaced Microsoft.EnterpriseManagement.DataAccessService.Core.dll with the 1807 version and it seems to have fixed this. Have you seen this on other SCOM 2019 installs?

  13. Zikki

    WE have successfully completed UR6 using your steps. Thanks for the same. I would like to know whether Unix and Linux agent update is mandatory? We have version 7.6.1064 and 7.6.1076 installed. Please guide. Also, in SCOM management -> SCOM Agents I’m not able to see 2016 UR6 anywhere mentioned. What could be the reason.

    • Kevin Holman

      If you use Unix/Linux monitoring, then yes – you need to update everything. If you don’t use SCOM to monitor UNIX/Linux systems, then no.

      I don’t know why your SCOM management view doesn’t show UR6. Either you are not keeping that MP up to date, or your agents didn’t get updated. You need to troubleshoot this.

  14. Rabih

    Concerning the: Incorrect syntax near the keyword ‘CREATE’, it’s still occurring on SCOM 2019, is there anything i can do to fix it?

  15. Pingback:OpsMgr 2016 – QuickStart Deployment Guide - Kevin Holman's Blog

  16. Artūras Radzys

    Hi Kevin,

    bumped into one issue after doing management pack cleanup and was wondering that you have seen this before. Using SCOM 2019 UR1. After cleanup I’m no longer able to reset monitors health from health explorer. Mangement server reports event ID 26319 “An exception was thrown while processing SubmitTasks for session ID uuid:e992410a-9875-4643-9d5f-1f9629d5831a;id=20653.
    Exception message: The creator of this fault did not specify a Reason.
    Full Exception: System.ServiceModel.FaultException`1[Microsoft.EnterpriseManagement.Common.UnauthorizedAccessEnterpriseManagementException]: The creator of this fault did not specify a Reason. (Fault Detail is equal to Microsoft.EnterpriseManagement.Common.UnauthorizedAccessEnterpriseManagementException: The overrides provided for the task with id 4737ca24-843e-03bb-5c26-7c5336851d07 were invalid.).”

    maybe you have some ideas?

Leave a Reply

Your email address will not be published.