Menu Close

UR7 for SCOM 2016 – Step by Step

image

KB Article for OpsMgr: https://support.microsoft.com/en-us/help/4492182/update-rollup-7-for-system-center-2016-operations-manager

Download catalog site: http://www.catalog.update.microsoft.com/Search.aspx?q=4492182

Updated UNIX/Linux Management Packs: https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page: https://kevinholman.com/2009/01/27/which-scom-hotfixes-should-i-apply/

Key fixes:

  • Improved the performance of SCOM console in listing the groups.
  • Users of a scoped group are not able to use the Console.  (this resolves the bug introduced in UR6)
  • Fix: Operations Manager grooms out the alert history only on an alert closure.
  • Fixed:  Windows Computer Property “NetbiosDomainName” is not discovered properly.   (this was a REALLY old bug…. finally!!!!)
  • Fixed the formatting issue with the output for the task ‘Top10 CPU Processes’ when using Windows Management Infrastructure (MI) APIs.
  • Fixed:  Agents by Health State report shows duplicate agent names.
  • Fixed: Cannot use SQLOleDB.dll to probe databases like Oracle/MySQL.
  • Fixed an issue that prevented addition of a group in the Storage Spaces Direct 2016 management pack dashboard.
  • SCOM Network Device Re-Discovery now probes for SNMP V3 devices too.
  • Fixed re-registering for SNMP Traps in the Proxy Management Server.
  • SCOM console crashes while trying to connect to Azure Log Analytics and Azure Monitor.
  • Linux agent is not able to get the correct version and port details for JBoss EAP 7.1.
  • An issue that lead to creation of multiple empty temp files in the /tmp directory of Linux servers has been fixed.
  • XPlat agent now supports monitoring of SUSE-11 SP4 platform with Security Module installed on it for TLS 1.2 compliance.
  • Fixed an issue that caused the corruption of /etc/login.cfg file on AIX 7 machines during install/upgrade of the agent.
  • AIX Agent is now transitioned to 64-bit package to accommodate more stack and heap space if needed to avoid any stack/heap overflow which occasionally leads to heartbeat failure.
  • Free memory calculation accommodated appropriately on RHEL-7 platform.

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available.

 

Let’s get started

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management Servers
    • Audit Collection Server (ACS)
    • Web Console Servers
    • Gateway Servers
    • Operations Console Servers
    • Reporting Server
  2. Apply SQL scripts
  3. Manually import the management packs
  4. Apply Agent Updates
  5. Update Nano Agents
  6. Update Unix/Linux MP’s and Agents

WHOA Nelly

Before we get started – there are some configuration changes necessary in SQL to support UR6 and later.  UR6 (and later) has some fixes for Scheduled Maintenance, and these require some rights configuration FIRST – BEFORE applying UR6 or later.  Otherwise you might run into some trouble.  If you applied UR6 and already made these changes, then this step should just be verifying they are correct.

Essentially, you need to:

  1. Open SQL Management studio
  2. Connect to the SQL instance that hosts your OperationsManager database
  3. Select your SQL login for the Management Server Action account, choose properties.
  4. Select “User Mappings”, and Add/verify a user mapping to the MSDB database.
  5. In this user mapping – select db_owner, public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

image

  1. Now, open the properties for the SQL login for the SDK/DAS account.
  2. Select “User Mappings”, and Add/verify a user mapping to the MSDB database (you may already have one).
  3. In this user mapping – select db_owner, public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

image

Note:   If you do not configure these rights, you might see failures in the SCOM logs after applying UR6 (or later) about not being able to synchronize scheduled maintenance jobs, or you might see issues with creating new Schedule Maintenance failing, locking up the console, or even creating large numbers of duplicate jobs.

 

Management Servers

image

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, and I always recommend a manual installation on management servers, I DO NOT recommend ever using Windows Update.  My first management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

My first server is a Management Server, Web Console server, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but you MIGHT see a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 7 Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute:

image

Next up – run the Web Console update:

image

This runs much faster.   A quick file spot check:

image

Lastly – install the Console Update (make sure your console is closed):

image

A quick file spot check:

image

Or help/about in the console:

image

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

 

Updating ACS (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

image

image

Note the above image states “Operations Manager 2012”.  This is a known issue. Ignore it.

Updated files:

image

 

Updating Gateways:

image

Generally I can use Windows Update or manual installation.  I will proceed with manual:

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:

image

 

Apply the SQL Scripts

image

In the path on your management servers, where you installed/extracted the update, there are TWO SQL script files.  One for the Operations Database, and one for the Warehouse Database.

image

%SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment or clean install)

Operations Database update:

First – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation. 

I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Data Warehouse database update:

Next – let’s run the script to update the OperationsManagerDW (Warehouse) database.  Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file (UR_Datawarehouse.sql).  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

image

 

Manually import the management packs

image

There are 36 management packs in this update!  Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language. 

This is the initial import list:

image

image

What NOT to import:

The Advisor MP’s are only needed if you are connecting your SCOM environment to Microsoft Operations Management Suite / Log Analytics cloud service, which is very rare (Previously known as Advisor, and Operations Insights)

DON’T import ALL the languages – ONLY ENU, or any other languages you might require.

The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (very rare)

The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM.

So I remove what I don’t want or need – and I have this:

image

#Note: If the “Install” button is greyed out – this means you might already have one or more of these MP’s with the same version installed.  Find it by scrolling through each one, the console will tell you if you already have the same version.

 

Update Agents

image

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.

You can approve these – which will result in a success or failure message once complete:

image

You normally can verify the PatchLevel by going into the console and opening the view at: Monitoring > Operations Manager > Agent Details > Agents by Version

image

I *strongly* recommend you take a look at this community MP, which helps see the “REAL” agent number in the Administration –> Agent Managed view console:

https://kevinholman.com/2017/02/26/scom-agent-version-addendum-management-pack/

image

And my SCOM Management Group Management mp (updated for UR7), which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

image

 

Update UNIX/Linux MPs and Agents

image

The UNIX/Linux MP’s and agents at the time of this article publishing have been updated for UR7.

You can get the current Unix/Linux MP updates here: https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR7 is 7.6.1092.0  – and includes agents with version 1.6.2-343

Make sure you download the correct version for your SCOM deployment version:

image

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

image

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:

image

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

You should see the new files dropped with new timestamps:

image

Now you can deploy the Linux agent updates:

image

image

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

image

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://kevinholman.com/2016/11/11/monitoring-unix-linux-with-opsmgr-2016/

 

Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

Known Issues:

1. The “Operations Manager” Event log on SCOM servers is forced back to the default value of 16MB after you apply UR7.  If you customized this value on Management Servers (and I recommend doing this) to a larger value such as 60-100MB, this value will be set back to 16MB.  This will result in a Management Server event log that is too small for best practices, and you should set this back to your previously modified value.

2.  Customized registry values might be wiped out by UR7 and reset to default values.  For instance, one of the things I change on SCOM *Management servers*, is the registry value at “HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Persistence Checkpoint Depth Maximum”.  This is forced back to a default value of 20971520 after you apply UR7.  I recommend this value be set to 104857600 on SCOM management servers.  There could be others.  If you customize Registry settings on SCOM Agents or Management Servers, you should verify them after applying UR7 as part of your testing, and make a plan to automate fixing them if necessary.

3.  UR7 will overwrite the registry path at “HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\InstallDirectory” and modify this path to the C: drive, even when the SCOM Agent is installed to some other drive letter, such as the E: drive.  This can break some automations.  You should correct this registry entry via MP automation if this affects you.

4.  The ACS update shows “Operations Manager 2012” in the UI but is actually for SCOM 2016.

13 Comments

  1. Pingback:SCOM 2016: UR7 is Live | OpsMan

    • Kevin Holman

      While #1 is not good…. it isnt too bad, I mean – at least it is an easy fix on a small number of servers. I wrote an MP that monitors for this and has a recovery to even set it back – that’s how I found out this was even happening!

      #3 has actually been around a LONG time, in previous UR’s. I just forget to document it. I only have ever had one customer that installed the SCOM agent in a customized path on the non-C drive…. and most of the time people affected by this bug would never even know. However, I write some automations in my SCOM.Management MP which look for the agent path and get it from this registry location, which breaks my discoveries and throws script errors for file not found. I think it is only agents.

  2. Bob Cornelissen

    To set the #2 known issue back:

    reg add “HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters” /v “Persistence Checkpoint Depth Maximum” /t REG_DWORD /d 104857600 /f

    And to set the #1 known issue to another value (I have just added a 0 at the end to end up with a 150 MB SCOM log this way on management servers):

    reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\EventLog\Operations Manager” /v “MaxSize” /t REG_DWORD /d 157286400 /f

    I just run these two command now after the UR7 on management server.

  3. Den

    Hi Kevin,

    After update from SCOM 2012 R2 to 2016 UR6 we have a problem with web console: “500 – Internal server error”.
    Has this bug been fixed in UR7?
    Thank you

    • Kevin Holman

      Yes – that should be related to the:

      Key fixes:

      Users of a scoped group are not able to use the Console. (this resolves the bug introduced in UR6)

    • Kevin Holman

      The KB is misleading. You do NOT need to apply ANY other UR before going straight to UR7. The KB states that ONLY because the “uninstall” of UR’s didnt work until UR2. Therefore – the product team felt they should recommend applying UR2 (or later) first, so that any subsequent UR could be uninstalled. I disagree, it is a wasted step, as we don’t normally EVER uninstall a UR…. and if you need to uninstall a UR, you will be able to do so after the next UR releases. I recommended they remove that from the KB because it only causes confusion.

      I was as clear as I can be:

      NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available.

  4. Pingback:System Center Mayıs 2019 Bülten – Sertaç Topal

  5. Paul

    Hi Kevin,

    When importing the updated management packs included as part of UR7 would it be necessary to export, update (the override mp version number) and re-import the associated override management packs? OR is this not necessary?

    For example I have a custom overrides MP “System Center Core Monitoring_OVERRIDES”. When importing UR7 MP’s I see “System Center Core Monitoring” MP is updated. Do I subsequently need to export, update the version number (of the overrides MP) and re-import my overrides MP?

    The reason why i am confussed on this issue is because i have read under management pack lifecycle that:

    “When a management pack is updated, update the corresponding _Overrides.xml file with the new version number.”

    https://docs.microsoft.com/en-us/system-center/scom/manage-mp-lifecycle?view=sc-om-2019

    >
    I don’t fully understand that statement from MS, however my thinking is that actually I only need to update the overrides MP version number if i am exporting, changing and re-importing the OVERRIDES MP(?). Simply importing a sealed updated MP as in UR7 does not require my associated overrides mp version number to be updated.

    I would greatly appreciate your clarification here as i am currently planning our UR7 deployment.

    • Kevin Holman

      I would never mess with updating version numbers on override MP’s, and disagree with that position being part of a beneficial MP lifecycle.

Leave a Reply

Your email address will not be published. Required fields are marked *