UR10 for SCOM 2016 – Step by Step

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/4580254/update-rollup-10-for-system-center-2016-operations-manager

Download catalog site:  http://www.catalog.update.microsoft.com/Search.aspx?q=4580254

Updated UNIX/Linux Management Packs:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page:  https://kevinholman.com/2009/01/27/which-scom-hotfixes-should-i-apply/

Key fixes:

• The cmdlet Export-SCOMEffectiveMonitoringConfiguration has been fixed to give a correct summary of the applicable monitors, rules and overrides on an object.
• VB scripts for partition and grooming, calculate operations manager free space and detecting duplicate agent will now run without failure even if SNAC or MSOLEDBSQL are not installed.
• The issue regarding the TLS 1.2 compatibility in the OleDB module has been fixed. It is no longer mandatory for the provider element to be the first reference in the connection string.
• Performance improvement: Added “Recompile” hint to the stored procedures “p_SelectForTypeCache” and “p_SelectForNewTypeCache” that run frequently on SCOM DB.
• The security issue regarding reverse tabnabbing has been fixed in the operations manager web console.
• Fixed the Cross-site Scripting (XSS) related security issue in the operations manager web console.
• Fix has been provided for when the monitor erroneously turned critical due to the URL module incorrectly parsing the charset header value.
• The exception which blocked further progress when the user attempted to configure web application availability monitoring has been fixed.
• Management Pack Import is now compatible for SCOM 2007 –> SCOM 2016 Upgrade version when upgraded directly or Indirectly.
• Quarterly report end date will be shown correctly for the first quarter when the “From” field is selected as “First day of previous quarter” and “To” field is selected as “Last day of previous quarter”.
• Reports have been fixed to not show objects which have been deleted before the selected start time.
• The  .NET API  issue regarding scheduling reports via the schedule management wizard has now been fixed.

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available.

 

Let’s get started:

From reading the KB article – the order of operations is:

1. Install the update rollup package on the following server infrastructure:
   • Management Servers
   • Audit Collection Server (ACS)
   • Web Console Servers
   • Gateway Servers
   • Operations Console Servers
   • Reporting Server
2. Apply SQL scripts
3. Manually import the management packs
4. Apply Agent Updates
5. Update Unix/Linux MP’s and Agents

 

Prerequisites:

Before we get started – you should verify that your SQL rights are correct to support Scheduled Maintenance Mode.  When SCOM 2016 shipped this was not configured, and subsequent updates in UR1, UR6, and UR8 changed the way this works.   Scheduled MM still has a dependency to the SQL Agent, to calculate the next run time of a schedule.  However, the schedules themselves are now contained in the OperationsManager database to better support AlwaysOn and Database move scenarios.

Essentially, you need to:

1. Open SQL Management studio
2. Connect to the SQL instance that hosts your OperationsManager database
3. Select your SQL login for the Management Server Action account, choose properties.
4. Select “User Mappings”, and Add/verify a user mapping to the MSDB database.
5. In this user mapping – select public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

1. Now, open the properties for the SQL login for the SDK/DAS account.
2. Select “User Mappings”, and Add/verify a user mapping to the MSDB database (you may already have one).
3. In this user mapping – select public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

Note:   If you have never configured these rights, you might see failures in the SCOM logs about not being able to synchronize scheduled maintenance jobs, or you might see issues with creating new Schedule Maintenance failing, locking up the console, or even creating large numbers of duplicate jobs.

***Important:  Repeat this on any SQL instances that are part of a SQL AlwaysOn Availability Group for your OperationsManager database, if applicable.

 

Install Microsoft OLE DB Driver for SQL Server

Starting with UR9 we included support for the MSOLEDB driver for SQL server.  This is the currently supported method to enable TLS 1.2 enforcement for SCOM communications with SQL server.  HOWEVER, even if you are not using TLS 1.2, it is recommended to install MSOLEDB 18.5 to prepare for a TLS 1.2 enforced enforcement.  Read more here: Microsoft OLE DB Driver for SQL Server  This driver should be installed on all SCOM Management Servers and Web Console servers, as they communicate directly with a SQL database that might become enforced for TLS 1.2 only.

Download Microsoft OLE DB Driver for SQL Server

 

Management Servers:

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, and I always recommend a manual installation on management servers, I DO NOT recommend ever using Windows Update.  My first management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

My first server is a Management Server, Web Console server, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

This launches a UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but you MIGHT see a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Level:         Information
Description:  Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 10 Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute:

Next up – run the Web Console update:

Lastly – install the Console Update (make sure your console is closed):

You can check help/about in the console:

Additional Management Servers:

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

ACS Update: (Audit Collection Services)

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

Note the above image states “Operations Manager 2012”.  This is a known issue. Ignore it.

Updated files:

Updating Gateways:

Generally I can use Windows Update or manual installation.  I will proceed with manual:

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

***NOTE:  You can delete any older UR update files from the \AgentManagement directories on Management Servers and Gateways.  The UR’s do not clean these up and they provide no purpose for being present any longer.

Updating Reporting:

On your server that hosts the SCOM Reporting role, run the update:

Apply the SQL Scripts:

In the path on your management servers, where you installed/extracted the update, there are TWO SQL script files.  One for the Operations Database, and one for the Warehouse Database.

• %SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups
• (note – your path may vary slightly depending on if you have an upgraded environment or clean install)

Operations Database update:

First – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation. 

I have had customers state this takes from a few seconds to a few minutes.  

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  If your scripts are being blocked, you MIGHT have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run, but this should be very rare in SCOM 2016 and later.

Data Warehouse database update:

Next – let’s run the script to update the OperationsManagerDW (Warehouse) database.  Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file (UR_Datawarehouse.sql).  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

Manually import the management packs:

There are 40 management packs in this update!  Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

• \Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language. 

This is the initial import list:

What NOT to import:

• The Advisor MP’s are only needed if you are connecting your SCOM environment to Microsoft Operations Management Suite / Log Analytics cloud service, which is rare (Previously known as Advisor, and Operations Insights)
• DON’T import ALL the languages – ONLY ENU, or any other languages you might require.
• The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (very rare)
• The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM. (very rare)

So I remove what I don’t want or need – and I have this:

#Note: If the “Install” button is greyed out – this means you might already have one or more of these MP’s with the same version installed.  Find it by scrolling through each one, the console will tell you if you already have the same version.  In my case, I already had the Microsoft Telemetry Pack imported, and had to remove this one from my list:

 

Update Agents:

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

If your agents are not placed into pending management – the most common causes of this are:

• You didn’t run the server update from an elevated command prompt
• You didn’t run the server role update with an account that has SCOM Admin rights AND SysAdmin SQL rights to the instance hosting the OperationsManager database.
• You used Windows Update to supply the server role update rollup.
• Your agents are nor set to Remotely Manageable = Yes, such as with manually installed agents.

You can approve these – which will result in a success or failure message once complete:

You normally can verify the PatchLevel by going into the console and opening the view at: Monitoring > Operations Manager > Agent Details > Agents by Version

I *strongly* recommend you take a look at this community MP, which helps see the “REAL” agent number in the Administration –> Agent Managed view console:

https://kevinholman.com/2017/02/26/scom-agent-version-addendum-management-pack/

And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

 

Update UNIX/Linux MPs and Agents:

You can get the current Unix/Linux MP updates here: https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR10 is 7.6.1092.0  – and includes agents with version 1.6.2-343

Make sure you download the correct version for your SCOM deployment version:

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

You should see the new files dropped with new timestamps:

Now you can deploy the Linux agent updates:

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://kevinholman.com/2016/11/11/monitoring-unix-linux-with-opsmgr-2016/

Update the remaining deployed consoles:

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

Known Issues:

1.  UR10 will overwrite the registry path at “HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\InstallDirectory” and modify this path to the C: drive, even when the SCOM Server Role or Agent is installed to some other drive letter, such as the E: drive.  This can break some automations.  You should correct this registry entry via MP automation if this affects you.

2.  The ACS update shows “Operations Manager 2012” in the UI but is actually for SCOM 2016.