Menu Close

UR9 for SCOM 2016 – Step by Step

image

KB Article for OpsMgr: https://support.microsoft.com/en-us/help/4546986/update-rollup-9-for-system-center-2016-operations-manager

Download catalog site: http://www.catalog.update.microsoft.com/Search.aspx?q=4546986

Updated UNIX/Linux Management Packs: https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page: https://kevinholman.com/2009/01/27/which-scom-hotfixes-should-i-apply/

Key fixes:

  • Fixed: On Windows Computer View of SCOM Console, Queries are optimized for better performance.
  • Fixed: We have updated stored procedure to handle Alert Source Path Names which are greater than 512 characters in length.  Previously this could cause data truncation and console crashes.
  • Fixed: Made changes to the IP address parameter to support both IPv4 and IPv6 windows cluster.
  • Fixed: Fixed the issue with conversion of data smaller than 0.01 which was generated by Rule or Monitor.  The data was transformed to a wrong (big) value on systems with OS locale language which has comma (“,”) instead of dot (“.”) as decimal format.
  • Fixed: Previously In UR8, we added support for customizing Memory Trimming on the Healthservice.exe process.  The primary change in UR9 was adding this same capability to the MonitoringHost.exe processes.  Description: In a scenario where SCOM monitors 100s of virtual machines hosted on a single Hyper-V server; every hour the MonitoringHost.exe of each Virtual machine will write into the VM page file simultaneously.  Due to this concurrent paging, every hour disk I/O spikes.  HealthService.exe and MonitoringHost.exe will continue to have Memory Trimming enabled by default on an hourly schedule.   However, a new registry control option is provided to disable the memory trimming or control the duration.

Key:  HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\MonitoringHost\MemoryTrimming

REG_DWORD Decimal Value:  Enable
SCOM default existing registry value:   (not present)
SCOM default value in code = 1
Enable = 0 (trimming is disabled)  Enable = 1 (trimming is enabled

REG_DWORD Decimal Value:  DelayInSeconds
Time period in seconds the agent waits after initialization to start trimming
SCOM default existing registry value:   (not present)
SCOM default value in code = 120

REG_DWORD Decimal Value:  PeriodInSeconds
Recurring period in seconds at which the working set should be trimmed
SCOM default existing registry value:   (not present)
SCOM default value in code = 3600
Minimum value = 3600  Maximum value = 4294967295

  • Fixed: Downtime duration will be calculated correctly for non-US time format in Downtime Report.
  • Fixed: fn_MPVersionDependentId creates an ID for each installed MP. The ID is a SHA1 hash of string “MPName, Version, Schema”. When the “Schema” part is omitted, it generates different hash which doesn’t match with the expected value. In the fix, “Schema=2” part is re-added in fn_MPVersionDependentId.
  • Fixed: We have made sure that correct end date for Availability, Health and Performance Details are shown in reports.
  • Enabled support for MSOLEDBSQL driver so that users may move from SQL Server Native Client. It is recommended to install Microsoft OLE DB Driver for SQL Server (MSOLEDBSQL) to enable support for OLE DB Driver.  This is most important to understand for TLS 1.2 enforced environments.  Since SQL Native Client is deprecated and not supported for SQL 2016 and later, MSOLEDBSQL is the supported provider for access to SQL.  Read more here: LINK

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available.

 

Let’s get started:

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management Servers
    • Audit Collection Server (ACS)
    • Web Console Servers
    • Gateway Servers
    • Operations Console Servers
    • Reporting Server
  2. Apply SQL scripts
  3. Manually import the management packs
  4. Apply Agent Updates
  5. Update Unix/Linux MP’s and Agents

 

 

Prerequisites:

Before we get started – you should verify that your SQL rights are correct to support Scheduled Maintenance Mode.  When SCOM 2016 shipped this was not configured, and subsequent updates in UR1, UR6, and UR8 changed the way this works.   Scheduled MM still has a dependency to the SQL Agent, to calculate the next run time of a schedule.  However, the schedules themselves are now contained in the OperationsManager database to better support AlwaysOn and Database move scenarios.

Essentially, you need to:

  1. Open SQL Management studio
  2. Connect to the SQL instance that hosts your OperationsManager database
  3. Select your SQL login for the Management Server Action account, choose properties.
  4. Select “User Mappings”, and Add/verify a user mapping to the MSDB database.
  5. In this user mapping – select public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

image

  1. Now, open the properties for the SQL login for the SDK/DAS account.
  2. Select “User Mappings”, and Add/verify a user mapping to the MSDB database (you may already have one).
  3. In this user mapping – select public, SQLAgentOperatorRole, SQLAgentReaderRole, SQLAgentUserRole

image

Note:   If you have never configured these rights, you might see failures in the SCOM logs about not being able to synchronize scheduled maintenance jobs, or you might see issues with creating new Schedule Maintenance failing, locking up the console, or even creating large numbers of duplicate jobs.

***Important:  Repeat this on any SQL instances that are part of a SQL AlwaysOn Availability Group for your OperationsManager database, if applicable.

 

Install Microsoft OLE DB Driver for SQL Server

UR9 includes support for the MSOLEDB driver for SQL server.  This is the currently supported method to enable TLS 1.2 enforcement for SCOM communications with SQL server.  HOWEVER, even if you are not using TLS 1.2, you MUST install either SQL Native Client (SNAC) 11 (or later) – OR – MSOLEDB 18.3 or later as a PREREQUISITE to UR9.  This is required, because many scripts will start to fail if you don’t.  I recommend choosing MSOLEDB over SNAC as that is the currently supported driver.  Read more here: LINK

Download Microsoft OLE DB Driver for SQL Server

 

 

 

Management Servers:

image

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, and I always recommend a manual installation on management servers, I DO NOT recommend ever using Windows Update.  My first management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

My first server is a Management Server, Web Console server, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but you MIGHT see a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Level:         Information
Description:  Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 9 Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute:

image

Next up – run the Web Console update:

image

Lastly – install the Console Update (make sure your console is closed):

image

You can check help/about in the console:

image

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

 

ACS Update: (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that:

image

image

Note the above image states “Operations Manager 2012”.  This is a known issue. Ignore it.

Updated files:

image

 

Updating Gateways:

image

Generally I can use Windows Update or manual installation.  I will proceed with manual:

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:

image

 

Apply the SQL Scripts:

image

In the path on your management servers, where you installed/extracted the update, there are TWO SQL script files.  One for the Operations Database, and one for the Warehouse Database.

image

  • %SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups
  • (note – your path may vary slightly depending on if you have an upgraded environment or clean install)

Operations Database update:

First – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation. 

I have had customers state this takes from a few seconds to a few minutes.  

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  If your scripts are being blocked, you MIGHT have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run, but this should be very rare in SCOM 2016 and later.

Data Warehouse database update:

Next – let’s run the script to update the OperationsManagerDW (Warehouse) database.  Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file (UR_Datawarehouse.sql).  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

image

 

Manually import the management packs:

image

There are 37 management packs in this update!  Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

  • \Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language. 

This is the initial import list:

image

image

image

What NOT to import:

  • The Advisor MP’s are only needed if you are connecting your SCOM environment to Microsoft Operations Management Suite / Log Analytics cloud service, which is rare (Previously known as Advisor, and Operations Insights)
  • DON’T import ALL the languages – ONLY ENU, or any other languages you might require.
  • The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (very rare)
  • The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM. (very rare)

So I remove what I don’t want or need – and I have this:

image

#Note: If the “Install” button is greyed out – this means you might already have one or more of these MP’s with the same version installed.  Find it by scrolling through each one, the console will tell you if you already have the same version.

 

Update Agents:

image

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

image

If your agents are not placed into pending management – the most common causes of this are:

  • You didn’t run the server update from an elevated command prompt
  • Your agents are nor set to , or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.

You can approve these – which will result in a success or failure message once complete:

image

You normally can verify the PatchLevel by going into the console and opening the view at: Monitoring > Operations Manager > Agent Details > Agents by Version

image

I *strongly* recommend you take a look at this community MP, which helps see the “REAL” agent number in the Administration –> Agent Managed view console:

https://kevinholman.com/2017/02/26/scom-agent-version-addendum-management-pack/

image

And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

image

 

Update UNIX/Linux MPs and Agents:

image

You can get the current Unix/Linux MP updates here: https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR9 is 7.6.1092.0  – and includes agents with version 1.6.2-343

Make sure you download the correct version for your SCOM deployment version:

image

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

image

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:

image

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

You should see the new files dropped with new timestamps:

image

Now you can deploy the Linux agent updates:

image

image

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

image

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://kevinholman.com/2016/11/11/monitoring-unix-linux-with-opsmgr-2016/

 

Update the remaining deployed consoles:

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

Known Issues:

image

1. The “Operations Manager” Event log on SCOM servers is forced back to the default value of 16MB after you apply UR9.  If you customized this value on Management Servers (and I recommend doing this) to a larger value such as 60-100MB, this value will be set back to 16MB.  This will result in a Management Server event log that is too small for best practices, and you should set this back to your previously modified value.

2.  Customized registry values might be wiped out by UR9 and reset to default values.  For instance, one of the things I change on SCOM *Management servers*, is the registry value at “HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Persistence Checkpoint Depth Maximum”.  This is forced back to a default value of 20971520 after you apply the Update Rollup.  I recommend this value be set to 104857600 on SCOM management servers.  There could be others.  If you customize Registry settings on SCOM Agents or Management Servers, you should verify them after applying any Update Rollup as part of your testing, and make a plan to automate fixing them if necessary.

3.  UR9 will overwrite the registry path at “HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\InstallDirectory” and modify this path to the C: drive, even when the SCOM Agent is installed to some other drive letter, such as the E: drive.  This can break some automations.  You should correct this registry entry via MP automation if this affects you.

4.  The ACS update shows “Operations Manager 2012” in the UI but is actually for SCOM 2016.

5.  UR9 requires SQL Native Client 11, or MSOLEDBSQL 18.  See links above in the Prerequisite section for downloads.  If you do not have one of these installed, once you apply UR9 you will see serious script failure issues in your Management Server event logs:

Log Name:      Operations Manager
Source:        Health Service Modules
Event ID:      21406
Level:         Warning
Description:  The process started at 12:11:12 PM failed to create System.PropertyBagData. Errors found in output:

C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\Health Service State\Monitoring Host Temporary Files 24\2037\GetOpsMgrDBPercentFreeSpace.vbs(131, 5) Microsoft VBScript runtime error: Type mismatch: ‘[string: “BB9”]’
Command executed:    “C:\Windows\system32\cscript.exe” /nologo “GetOpsMgrDBPercentFreeSpace.vbs” omtstsql1\inst1 OperationsManager
Working Directory:    C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\Health Service State\Monitoring Host Temporary Files 24\2037\

One or more workflows were affected by this.

Workflow name: Microsoft.SystemCenter.OpsMgrDBPercentFreeSpaceMonitor
Instance name: All Management Servers Resource Pool DB Watcher
Instance ID: {562FB615-2AB8-4BB6-9244-784AB8C67693}
Management group: SCOMTST

 

Log Name:      Operations Manager
Source:        Health Service Script
Event ID:      100
Level:         Error
Description:  DetectDuplicateRelAgnToSrvMonitor.vbs : Script executed with Error Number: 3001 Error Details: Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another.

 

Log Name:      Operations Manager
Source:        Health Service Script
Event ID:      1199
Level:         Error
Description:  GetOpsMgrDWDBWatcherDiscovery.vbs : Error returning discovery data. code = 3001

 

Log Name:      Operations Manager
Source:        Health Service Script
Event ID:      100
Level:         Information
Description:  PartitioningAndGroomingMonitor.vbs : Error Number: 3001 Error Details: Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another.

16 Comments

  1. Dinesh Kumar

    Hi Kevin , Once again a great document .
    We were waiting for RHEL 8 support from UR9 2016.
    But its not included.
    Any thoughts when will it be released.

    • Kevin Holman

      I don’t know. This is somewhat new to me and I was surprised when I saw this in UR9 and had to ask the PG for the backstory. I would say the biggest reason would be to be “supported” but it’s probably not a big deal either way.

      • Steve

        Good afternoon Kevin –

        Along these lines, in our SCOM 2016 environment, we currently have SSMS 2016 (which comes with ODBC Driver 13 for SQL Server) installed on our management servers in addition to SQL Server Native Client 11 v2011.110.7001.00 which we installed separately believing it was required for TLS 1.2 support.

        Is there a way to tell which driver SCOM is using to connect to the DB? I was under the impression/thought it would use the ODBC driver as an alternative to the SQL Server Native Client 11?

        I see 1 instance in our Operations Manager log of one of the event IDs you mentioned above. However, I only see 1 instance of it and it was from a couple of days ago during the upgrade process to UR9. Since we have the required SQL Server Native Client 11 installed, is it possible that this one occurrence of the error was just related to the upgrade itself? I am assuming we would see the error reoccur if it was really a problem?

        Thank you!

        * Event ID 100
        * Description: DetectDuplicateRelAgnToSrvMonitor.vbs : Script executed with Error Number: 3001 Error Details: Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another

    • Kevin Holman

      I don’t know. It wasn’t in the list. That makes me think it is probably slated for UR2 if affected. But sometimes small fixes get swept into a UR and they don’t put them in the list.

  2. Mark Schlebaum

    Hi great post, but I have a question
    I upgraded the test environment from RTM version to UR9 in one step.
    When I check the agent version after the upgrade I see the following:
    Agent version is the Original 8.10918.0 but the patch list is System Center 2016 Operations Manager Update Rollup 9 Patch
    Your picture shows the correct version of the agent.
    When I remove a agent and install it from scratch with a discovery the version becomes 8.0.11049 with Patch List System Center 2016 Operations Manager Update Rollup 9 Patch

    Can you tell me if this is correct?

  3. Pingback:System Center Nisan 2020 Bülten – Sertaç Topal

  4. Atif Aman

    Hi Kevin,
    Another great document.

    I have a quick question?
    Does UR-9 replace legacy SQL MPs with new Agnostic MP?

    Thanks
    Atif Aman

  5. curtiss

    getting this same error after ur9 that i got after ur8, in two different environments. discussed in the comments on your step by step article for ur8.

    System Center Core Library could not be imported.

    If any management packs in the Import list are dependent on this management pack, the installation of the dependent management packs will fail.

    Database error. MPInfra_p_ManagementPackInstall failed with exception:
    [MP ID: 7cfc5cc0-ae0a-da4f-5ac2-d64540141a55][MP Version: 7.0.8437.16][MP PKT: 31bf3856ad364e35] Database error. MPInfra_p_ManagementPackInstall failed with exception:
    Invalid column name ‘DisplayName’.
    Transaction count after EXECUTE indicates a mismatching number of BEGIN and COMMIT statements. Previous count = 2, current count = 0.

    • Kevin Holman

      We are tracking this issue – it seems to come up in customer environments where they did in place upgrades. We have seen in SCOM 2016 post UR8 as well. I believe we only see in SCOM environments that began as SCOM 2007 and have been continuously upgraded. It is planned to be resolved in the next Update Rollup.

      If you are affected – anytime you import a MP with a Group definition or a Singleton class definition, this happens.

      This happens inside of p_ManagementPackFinishInstall here on these lines:

      SET @StatementForMT = N’UPDATE MT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’SET MT.DisplayName’ + N’ = LT.LTValue’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’FROM dbo.’ + @TableNameForMT + N’ MT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’JOIN dbo.LocalizedText LT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’ ON (MT.BaseManagedEntityId = LT.LTStringId AND LT.LanguageCode = dbo.fn_GetInstallationLocale() AND LT.LTStringType = 1)’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’WHERE MT.DisplayName <> LT.LTValue’ + NCHAR(13);

      The problem as said, is that in environments that were upgraded from 2007 R2, the column name is not be simply “DisplayName”, but in fact, it will be called “DisplayName_55270A70_AC47_C853_C617_236B0CFF9B4C” (exact same GUID all the time).

      Inside that stored proc, you’d have to edit it and do something like:

      DECLARE @DisplayNameColumnName VARCHAR(256)
      SELECT @DisplayNameColumnName = ColumnName
      FROM ManagedTypeProperty
      WHERE ManagedTypeId = dbo.fn_ManagedTypeId_SystemEntity()

      SET @StatementForMT = N’UPDATE MT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’SET MT.’ + @DisplayNameColumnName + N’ = LT.LTValue’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’FROM dbo.’ + @TableNameForMT + N’ MT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’JOIN dbo.LocalizedText LT’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’ ON (MT.BaseManagedEntityId = LT.LTStringId AND LT.LanguageCode = dbo.fn_GetInstallationLocale() AND LT.LTStringType = 1)’ + NCHAR(13);
      SET @StatementForMT = @StatementForMT + N’WHERE MT.’ + @DisplayNameColumnName + ‘ <> LT.LTValue’ + NCHAR(13);

      If you open a support case, they will guide any customer through this.

  6. Ørjan Landgraff Larsen

    “..users may move from SQL Server Native Client”
    Data warehouse database watcher and rollup dependencies went all Uninitialized after doing the switch – could be related to a deprecated monitor, or script logic that needs to be addressed for the “System Center Internal Library” management pack.?
    Example instancename: Microsoft.SystemCenter.OpsMgrDWWatchersGroup(Data Warehouse Database)

    Seems we are at-least two, with the same issue =) :
    https://community.squaredup.com/answers/question/a-word-of-caution-about-your-data-warehouse-health-monitor/

Leave a Reply

Your email address will not be published. Required fields are marked *