Menu Close

UR1 for SCOM 2019 – Step by Step

image

KB Article for OpsMgr: https://support.microsoft.com/en-us/help/4533415/update-rollup-1-for-system-center-operations-manager-2019

Download catalog site: http://www.catalog.update.microsoft.com/Search.aspx?q=4533415

Download for the NEW Simplified Management Server Update: HERE

Updated UNIX/Linux Management Packs: https://www.microsoft.com/en-us/download/details.aspx?id=58208

Recommended hotfix page: https://kevinholman.com/2009/01/27/which-scom-hotfixes-should-i-apply/

 

New Features:

  • Support for using Group Managed Service Accounts (gMSA).  LINK
  • Simplified management server patching – a new management server update process which will manage the SQL script updates and Management Pack imports automatically, resolving this common issue with inconsistent Update Rollup application.  LINK
  • Distro-Agnostic management pack for Linux.  New Linux platforms will not require distro-specific MP’s, making monitoring Linux platforms simpler.  LINK
  • Support for RHEL 8
  • Performance improvements in Linux for Heartbeat issues, separating Heartbeat from Performance processes.
  • Multi-Language installers.  Removing the need for language specific update files making the UR download process simpler.

Key fixes:

  • See the KB article.  There are too many fixes to list here!  LINK

 

NOTE:  I get this question every time we release an update rollup:  ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2019 and never applied an update rollup – you can go straight to the latest one available.

 
Let’s get started

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management Servers
    • Web Console Servers
    • Gateway Servers
    • Operations Console Servers
    • Reporting Server
  2. Apply Agent Updates
  3. Update Unix/Linux MP’s and Agents

 

 

Management Servers

image

Updates in SCOM 2019 have CHANGED.  There is a new process for updating management servers that differs from previous versions of SCOM.  Download the single file management server update, and this will ensure that your Management Server Role is updated, as well as any SQL updates, and Management Pack updates.

Download: LINK

High level install:  LINK

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I have 2 management servers  My first management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents.

image

 

Notice the new EXE file, and a MSP file exist for the Server update.  The EXE is the new simplified update file, but we included the older MSP for customers who want to continue to use the old process, or use silent installs for patching.  I will ONLY demonstrate and recommend the EXE file.

Once I have the EXE and MSP files, I am ready to start applying the update to each server by role.

***Note:  One of the changes in SCOM 2019 Update Rollups, is that you no longer need to have “Sysadmin” role level rights to SQL.  The SCOM Update Rollup simply updates SCOM, and then uses your existing RunAs accounts to deploy the updated SQL script files to modify the SQL databases.  You simply need to log into your SCOM management servers as a Local Administrator and SCOM Admin.

***Note:  There is ONE EXCEPTION to the above statement.  In order for agents to be placed into “Pending Management” view to allow approval of agent patching from the console, you must still have Sysadmin level rights to the SQL server hosting the OperationsManager database.  This flip of agents to pending is a process called on the management server, but it directly communicates with the database.  If you care about this feature, then you will still need Sysadmin to SQL as you did in previous updates.

 

My first server is a Management Server, Web Console server, and has the SCOM console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt.

I will start with KB4533415-AMD64-Server.exe

This is a self-extracting executable, that kicks off a simple update tool.  Accept the license terms, and click “Install”

This will update the management server role, update the databases with SQL scripts, and then import any Management Pack updates.

image

If you have an issue – you can review the setup logs:

Setup Log: C:\Users\<UserName>\Appdata\Local\SCOM\Logs

SQL Logs: <SCOM install directory>\Server\SQL Script for Update Rollups\SqlExceptions_{version}.log

MP Import Logs: <SCOM install directory>\Server\Management Packs for Update Rollups\ManualMPImport_{version}.log

***Note:  If you get an error here on the Management Pack Import – see “Known Issues” at the bottom of this post before continuing.

 

Next up – since this management server also runs a SCOM Web Console, I will run the Web Console update:     KB4533415-AMD64-WebConsole.msp

Lastly – install the Console Update (make sure your console is closed):   KB4533415-AMD64-Console.msp

You can reboot the server at this time if you were prompted to in order to complete the update.  If you were not prompted to, you do not need to.

 

 

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You should only patch one management server at a time to allow for graceful failover of agents and to keep resource pools stable.

You can use the same EXE file you used for the first management server.  The setup program will detect if the SQL scripts are already completed, and if the MP’s are already imported, and skip those if needed.

 

 

 

Updating Gateways:

image

Open an elevated command prompt, and run the update:   KB4533415-AMD64-Gateway.msp

The update launches a UI and quickly finishes.

 

I can spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

 

 

 

Updating Reporting:

image

On your server that hosts the SCOM Reporting role, run the update:   KB4533415-AMD64-Reporting.msp

 

 

 

Update Agents

image

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

***NOTE: For this to work, you MUST run the server update from an elevated command prompts, and the user account must be a local admin, SCOM Admin, and have Sysadmin SQL permissions to the instance hosting the OperationsManager database.  The Agents MUST have “Remotely Manageable” set to “Yes”.

You can approve these – which will result in a success or failure message once complete:

image

 

There is a new MP that shipped out of band, available for UR1, that will fix the long known issue that the Agent Version is not correctly displayed on Agent Managed screen.  You can download this new MP separately from the KB article and I am posting the link HERE.

image

 

Now we will show the “REAL” agent number in the Administration –> Agent Managed view console:

 

image

 

And my SCOM Management Group Management MP, which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://kevinholman.com/2017/05/09/scom-management-mp-making-a-scom-admins-life-a-little-easier/

image

 

 

 

Update UNIX/Linux MPs and Agents

image

You can get the current Unix/Linux MP updates here: https://www.microsoft.com/en-us/download/details.aspx?id=58208

If you have trouble getting the latest updated files, here is a direct LINK

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor.  Here is the FULL list:

image

In my environment – I only monitor RedHat and Universal Linux distributions, so this is my pared down list of MP’s to update:

image

 

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, and before you attempt to update your Linux Agents – verify the updated files are dropped at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents.   If they are not present, sometimes you must restart the Microsoft Monitoring Agent service on the management servers after an update.

 

You should see the new files dropped with new timestamps:

image

 

Now you can deploy the Linux agent updates:

image

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://kevinholman.com/2016/11/11/monitoring-unix-linux-with-opsmgr-2016/

 

 

Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

 

Verifying the update

There are new views in the SCOM console to help with this and make this process MUCH easier.  You do need to wait long enough for the discoveries to run in order for these to update the views. 

image

 

image

image

image

image

image

image

image

 

 

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, review the Management Group Health dashboard, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

 

Known Issues:

 image

1. You may see an error during the Management Import stage of the update, similar to this in the logs:

“System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> Microsoft.EnterpriseManagement.Common.ManagementPackException: Cannot load the management pack from the specified sealed assembly file.  Microsoft.EnterpriseManagement.Common.ManagementPackException: This assembly is not fully signed. Cannot verify the strong name signature of the file”

This fails due to a dependency on .NET 3.5 which is missing.  UR1 requires .NET 3.5 to be installed on the Management Server in order to attempt MP Import.  However, .NET 3.5 is not required to install nor support SCOM 2019.  This will be fixed in UR2 and we will not require .NET 3.5 anymore for Update Rollups.

You have two workarounds.

Workaround 1:  Simply ignore the error, continue patching, then MANUALLY import the management packs updating only the ones you already have imported, from \Program Files\Microsoft System Center\Operations Manager\Server\Management Packs for Update Rollups\

Workaround 2:  Install .NET 3.5 on the management server, and re-run the Update Rollup.

2.  In the Console, “Help > About” does not reflect that the update was applied to the Console.  UR1 does not update the version in Help/About like previous update rollups for SCOM did. 

You have two workarounds:

Workaround 1:  Use the Console views: Administration  > Operations Manager Products > Operations Consoles  to verify patch level.

Workaround 2:  Inspect the file version of: \Program Files\Microsoft System Center\Operations Manager\Console\Microsoft.MOM.UI.Components.dll to be 10.19.10311.0

3.  The “Operations Manager” Event log on SCOM servers is forced back to the default value of 16MB after you apply the Update Rollup

f you customized this value on Management Servers (and I recommend doing this) to a larger value such as 60-100MB, this value will be set back to 16MB.  This will result in a Management Server event log that is too small for best practices, and you should set this back to your previously modified value.

4.  Customized registry values might be wiped out by the Update Rollup and reset to default values. 

For instance, one of the things I change on SCOM *Management servers*, is the registry value at “HKLM\SYSTEM\CurrentControlSet\services\HealthService\Parameters\Persistence Checkpoint Depth Maximum”.  This is forced back to a default value of 20971520 after you apply the Update Rollup.  I recommend this value be set to 104857600 on SCOM management servers.  There could be others.  If you customize Registry settings on SCOM Agents or Management Servers, you should verify them after applying any Update Rollup as part of your testing, and make a plan to automate fixing them if necessary.  For more details, see https://kevinholman.com/2017/03/08/recommended-registry-tweaks-for-scom-2016-management-servers/

5.  The Update Rollup will overwrite the registry path at “HKLM\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\InstallDirectory” and modify this path to the C: drive, even when the SCOM Agent is installed to some other drive letter, such as the E: drive. 

This can break some automations that read this registry path to determine the agent file location.  You should correct this registry entry manually or via MP automation if this affects you.

29 Comments

    • Kevin Holman

      No it won’t! The UR’s will now only update an MP if you already have it imported, unless it is special… like the new MP which manages updating the SQL DBs.

  1. Thorsten Schneider

    We’ve run through the installation in our test environment and everything has worked ok. However, I’m confused by the version numbers:
    * The updated SCOM console shows version 10.19.10050.0 in Help/About which is the same as 2019 RTM. The “Operations Consoles” view in SCOM itself shows the correct version (10.19.10311.0)
    * An upgraded agent shows in “Add/Remove programs” still version 10.19.10014.0, while the DLLs have 10.19.10140.0.

    • Kevin Holman

      You are correct. I have reported to the PG that “Help/About” in the console doesn’t show the update and that customers would expect that. However, its good in the Admin section of the console…. but that doesnt help people who run the console on their non-managed desktops, they need an easy way to check their UR level. I hope they will fix this.

      I am not sure we ever updated add/remove programs agent version in any version of SCOM.

      • Thorsten Schneider

        Now that is really confusing:
        I’ve got a multi-homed agent between our test environment (2019 UR1) and our normal environment (2019 RTM). The agent has the UR1 patch installed and shows the new file versions (10.19.10140.0).

        In the updated test environment I see that new version and patch ONLY in the Administration / Agent view. The Monitoring / Operations Manager / Agents by Version still show 10.19.10014.0 and no patch installed).
        Interestingly enough the 2019 RTM environment shows that correct version and patch level in that view.

        Even worse:
        the same problem is in the Administration / Agent Managed view. 2019 UR1 shows the agent as 10014. So how should one find out which agents need to be updated ? That view would be my first choice to check and then to do a right-click Repair.

        Something seems to be broken here.

        • Kevin Holman

          You might be missing the step I called out above, where you need to import the out of band released MP, which fixes the “version” problem in the “Agent Managed” view. I got the product group to add that, and it will be included in UR2 and forward. It will show the correct agent version in “Agent Managed” so you can easily find/update agents that need it.

          On the “Agents by Version” view, it works fine, and shows the correct version and Patchlist shows UR1 as well.

  2. Max Friedrich

    Hi Kevin,
    thanks again for the very good post!

    In our testing enviroment i got the following error by installing UR1 on the first MP and didn’t find anything. The certificate status of the file is shown as “ok” at the server. Maybe you have an idea.

    “Exception thrown by custom action:
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> Microsoft.EnterpriseManagement.Common.ManagementPackException: Cannot load the management pack from the specified sealed assembly file: F:\Program Files\Microsoft System Center\Operations Manager\Server\Management Packs for Update Rollups\Microsoft.SystemCenter.2007.mp. —> Microsoft.EnterpriseManagement.Common.ManagementPackException: This assembly is not fully signed. Cannot verify the strong name signature of the file: F:\Program Files\Microsoft System Center\Operations Manager\Server\Management Packs for Update Rollups\Microsoft.SystemCenter.2007.mp”

    • Dev Ramdin

      Hi Kevin,

      I have the same issue and happen to be installing to a different drive as well.

      The existing version is 10.19.10050.0

    • frederik

      I am experiencing exactly the same issue with the .exe install method.
      When trying the old way, no issues occur, and the installation completes successfully.

  3. Anu

    Hello Kevin,

    While trying to install we are receiving the below error.

    KB4533415-AMD64-Server. An error occurred. Please view the setup log and try again later.

    When i checked the log files. i am able to see the below error. and am not able to find any mp is imported with today’s date
    [04:42:38]: Info: :Install Progress – (ImportManagementPacksFromPatch) Install Path = E:\Program Files\Microsoft System Center\Operations Manager\Server\
    [04:42:43]: Debug: :ApplyQuickFixEngineering: Return value was 1603. Check the log at C:\Users\\AppData\Local\Temp\KB4533415-AMD64-Server.msp.0.log for more detailed information.
    [04:42:43]: Error: :ApplyQfe: FAILED: We did not successfuly install QFE KB4533415-AMD64-Server.msp.
    [04:42:43]: Debug: :ProcessInstalls: Patcher returned error 1603:Fatal error during installation

    Error 3: -2147287038

    • Kevin Holman

      ANU, can you grab these logs, zip them up, and email them to me. We’d like to take a deeper look.

      kevin.holman@live.com

      Setup Log: C:\Users\\Appdata\Local\SCOM\Logs
      SQL Logs: \Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
      MP Import Logs:
      \Server\Management Packs for Update Rollups\ManualMPImport_{version}.log

  4. maca

    UR1 installation failed with Error code 1603. User is local administrator on Windows 2019 and does have sysadmin role in SQL2017. Could you please help?

    • Kevin Holman

      MACA, can you grab these logs, zip them up, and email them to me. We’d like to take a deeper look.

      kevin.holman@live.com

      Setup Log: C:\Users\\Appdata\Local\SCOM\Logs
      SQL Logs: \Server\SQL Script for Update Rollups\SqlExceptions_{version}.log
      MP Import Logs:
      \Server\Management Packs for Update Rollups\ManualMPImport_{version}.log

  5. MTEC

    Hi Kevin,

    If you guys are using Active Directory integration, do not update your agents just yet… the update strips the AD INT configuration leaving the agent not connected to any MGT group. MS have been informed.

  6. Mitesh

    Hi Guy’s
    ,
    If you are using Active Directory integration, do not update your agents just yet… the update strips the AD INT configuration leaving the agent not connected to any MGT group. MS have been informed, and currently investigating.

  7. Pingback:SCOM 2019 with UR1: Service Account Password Freedom at Last – blog.johnjoyner.net

  8. Raoul

    Simplified Management Server Update didn’t work for me. Like many others here I got a “KB4533415-AMD64-Server. An error occurred. Please view the setup log and try again later.”

    The old method works flawless.

  9. FredR

    I did not see anything about stopping services on management servers that have not been updated yet. Is that not necessary with the update rollup? If the services can stay up, it makes change approvals a bit easier. Thanks!

    • Kevin Holman

      That’s correct, there is not need to proactively stop services on all your SCOM management servers. You can update one server at a time and not require an outage anymore. Technically SCOM 2012R2 was the worst about this issue, getting the SQL scripts to deploy. However since SCOM 2016 the SQL scripts don’t make any big changes that get deadlocked. I carried over those recommendations from SCOM 2012R2, but most customers would not have an issue with SCOM 2016 either.

  10. Jonathan

    Hi Kevin,

    Since the update to UR1, my management server in Europe is not able to access the database with the error :
    System.Data.SqlClient.SqlException (0x80131904): Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

    I have a latency between 80 to 120 ms

    Note that my Database is in america.

    Regards

  11. Pingback:System Center Şubat 2020 Bülten – Sertaç Topal

Leave a Reply

Your email address will not be published. Required fields are marked *