There is already a very good deployment guide posted on Microsoft Docs here: https://docs.microsoft.com/en-us/system-center/scom/deploy-overview
The following article will cover a basic install of System Center Operations Manager 2019. The concept is to perform a limited deployment of OpsMgr, only utilizing as few servers as possible, but enough to demonstrate the roles and capabilities in SCOM. For this reason, this document will cover a deployment on 3 servers. A dedicated SQL server, and two management servers will be deployed. This will allow us to show the benefits of high availability for agent failover, and the highly available resource pool concepts. This is to be used as a template only, for a customer to implement as their own pilot or POC, or customized deployment guide. It is intended to be general in nature and will require the customer to modify it to suit their specific data and processes.
This also happens to be a very typical scenario for small environments for a production deployment. This is not a detailed architecture guide nor intended to displace the need for a complete and thorough design guide.
Server Names\Roles:
- OMSQL1 SQL Database Services, Reporting Services
- OMMS1 Management Server Role, Web Console Role, Console
- OMMS2 Management Server Role, Web Console Role, Console
Windows Server 2019 will be installed as the base OS for all platforms. All servers will be a member of the AD domain.
SQL 2019 CU11 will be the base standard for all database and SQL reporting services.
High Level Deployment Process:
1. In AD, create the following accounts and groups, according to your naming convention:
- DOMAIN\OMAA OM Server Action Account
- DOMAIN\OMDAS OM Config and Data Access Account
- DOMAIN\OMREAD OM Datawarehouse Reader Account
- DOMAIN\OMWRITE OM Datawarehouse Write Account
- DOMAIN\SQLSVC SQL Service Account
- DOMAIN\OMAdmins OM Administrators security group
2. Add the OMAA, OMDAS accounts to the “OMAdmins” global group.
3. Add the domain user accounts for yourself and your team to the “OMAdmins” group.
4. Install Windows Server 2019 to all server role servers.
5. Install Prerequisites and SQL 2019 CU11.
6. Install the Management Server and Database Components
7. Install the Reporting components.
8. Deploy Agents
9. Import Management packs
10. Set up security (roles and run-as accounts)
Prerequisites:
1. Install Windows Server 2019 to all Servers.
2. Join all servers to domain.
3. Install the Report Viewer controls to any server that will receive a SCOM console. Install them from here: DOWNLOAD. There is a prereq for the Report Viewer controls which is the “Microsoft System CLR Types for SQL Server 2014” (ENU\x64\SQLSysClrTypes.msi) available here: DOWNLOAD
4. OPTIONAL: If your organization enforces TLS 1.2, you must ensure the prerequisites for TLS 1.2 have been met on all Management Servers. TLS 1.2 Blog Post
5. Install all available Windows Updates to ensure the servers are patched and secure.
6. Add the “OMAdmins” domain global group to the Local Administrators group on each server.
7. Install IIS on any management server that will also host a web console:
Open PowerShell (as an administrator) and run the following:
Add-WindowsFeature NET-WCF-HTTP-Activation45,Web-Static-Content,Web-Default-Doc,Web-Dir-Browsing,Web-Http-Errors,Web-Http-Logging,Web-Request-Monitor,Web-Filtering,Web-Stat-Compression,Web-Mgmt-Console,Web-Metabase,Web-Asp-Net,Web-Windows-Auth –Restart
Note: The server needs to be restarted at this point, even if you are not prompted to do so. If you do not reboot, you will get false failures about prerequisites missing for ISAPI/CGI/ASP.net registration.
8. Install SQL 2019 CU11 to OMSQL1
- Setup is fairly straightforward. This document will not go into details and best practices for SQL configuration. Consult your DBA team to ensure your SQL deployment is configured for best practices according to your corporate standards.
- Run setup, choose Installation > New SQL Server stand-alone installation…
- When prompted for feature selection, install ALL of the following:
- Database Engine Services
- Full-Text and Semantic Extractions for Search
- Note: Reporting Services is not part of SQL DB Engine install. This is a separate download, we will cover that later.
- On the Instance configuration, choose a default instance, or a named instance. Default instances are fine for testing, labs, and production deployments. Production clustered instances of SQL will generally be a named instance. For the purposes of the POC, choose default instance to keep things simple.
- On the Server configuration screen, set SQL Server Agent to Automatic. You can accept the defaults for the service accounts, but I generally recommend using a Domain account for the service account. You should do whatever your DBA standards are here. Input the DOMAIN\SQLSVC account and password for SQL Server Agent, and SQL Server Database Engine.
- Check the box to grant Volume Maintenance Task to the service account for the DB engine. This will help performance when auto-grow is needed.
- On the Collation Tab – you can use the default which is SQL_Latin1_General_CP1_CI_AS
- On the Server Configuration tab – add your personal domain user account and/or a group you already have set up for SQL admins.
- On the Data Directories tab – set your drive letters correctly for your SQL databases, logs, TempDB, and backup.
- On the TempDB tab, notice that a high performance TempDB is being created, with a DB file for each CPU detected. Adjust sizes and autogrowth per your standards.
- Choose Install, and setup will complete.
- You will need to disable Windows Firewall on the SQL server, or make the necessary modifications to the firewall to allow all SQL traffic. See: Configure Firewall for SQL
- When you complete the installation – you might consider also downloading and installing SQL Server Management Studio Tools from: DOWNLOAD SQL Management Studio
9. Apply SQL 2019 CU11 (or whatever the latest Cumulative update available is). SCOM 2019 only supports SQL 2019 with CU8 or later and we STRONGLY recommend CU11 or later. At the time of this article being written, CU11 was the latest. Always install the latest Cumulative Update for SQL.
- There are no special instructions for CU11, simply apply the update accepting defaults.
- REBOOT the SQL server.
SCOM Step by step deployment guide:
1. Install the Management Server role on OMMS1.
- Log on using your personal domain user account that is a member of the OMAdmins group, and has “sysadmin” role level rights over the SQL instance.
- Run Setup.exe
- Click Install
- Select the following, and then click Next:
- Management Server
- Operations Console
- Web Console
- Accept or change the default install path and click Next.
- You might see an error from the Prerequisites here. If so – read each error and try to resolve it.
- On the Proceed with Setup screen – click Next.
- On the specify an installation screen – choose to create the first management server in a new management group. Give your management group a name. Don’t use any special or Unicode characters, just simple text. KEEP YOUR MANAGEMENT GROUP NAME SIMPLE, and don’t put version info in there. I typically use “OM1” as this is short, simple, and readily expandable in the future. Click Next.
- Accept the license. Next.
- On the Configure the Operational Database screen, enter in the name of your SQL database server name and instance. In my case this is “OMSQL1.domain.com”. Leave the port at default unless you are using a special custom fixed port. If necessary, change the database locations for the DB and log files. Leave the default size of 1000 MB for now. Click Next.
- On the Configure the Data Warehouse Database screen, enter in the name of your SQL database server name and instance. In my case this is “OMSQL1.domain.com”. Leave the port at default unless you are using a special custom fixed port. If necessary, change the database locations for the DB and log files. Leave the default size of 1000 MB for now. Click Next.
- On the Web Console screen, choose the Default Web Site, and leave SSL unchecked. If you have already set up SSL for your default website with a certificate, you can choose SSL, or configure this later. Click Next.
- On the Web Console authentication screen, choose Mixed authentication and click Next.
- On the accounts screen, change the accounts to Domain Account for ALL services, and enter in the unique DOMAIN\OMAA, DOMAIN\OMDAS, DOMAIN\OMREAD, DOMAIN\OMWRITE accounts we created previously. It is a best practice to use separate accounts for distinct roles in OpsMgr, although you can also just use the DOMAIN\OMDAS account for all SQL Database access roles to simplify your installation (Data Access, Reader, and Writer accounts). Click Next.
- On the Microsoft Update screen – choose OFF. Next.
- Click Install.
- Close when complete.
- The Management Server will be very busy (CPU) for several minutes after the installation completes. Before continuing it is best to give the Management Server time to complete all post install processes, complete discoveries, database sync and configuration, etc. 10 minutes is typically sufficient.
2. (Optional) Install the second Management Server on OMMS2.
- Log on using your domain user account that is a member of the OMAdmins group, and has System Administrator (SA) rights over the SQL instances.
- Run Setup.exe
- Click Install
- Select the following, and then click Next:
- Management Server
- Operations Console
- Web Console
- Accept or change the default install path and click Next.
- Resolve any issues with prerequisites, and click Next.
- Choose “Add a management server to an existing management group” and click Next.
- Accept the license terms and click Next.
- Input the servername\instance hosting the existing Ops DB. Select the correct database from the drop down and click Next.
- Accept the Default Web Site on the Web Console page and click Next.
- Use Mixed Authentication and click Next.
- On the accounts screen, choose Domain Account for ALL services, and enter in the unique DOMAIN\OMAA, DOMAIN\OMDAS accounts we created previously. Click Next.
- On the Diagnostic Data screen – click Next.
- On the Microsoft Update screen – choose OFF. Next.
- Click Install.
- Close when complete.
3. Install SCOM Reporting Role on the OMSQL1.
- Log on using your domain user account that is a member of the OMAdmins group, and has ”sysadmin” role level rights over the SQL instance.
- Install SQL 2019 Reporting Services
- Download SQL 2019 Reporting Services from DOWNLOAD
- Run SQLServerReportingServices.exe
- Select “Install Reporting Services”
- Input a product key for your license. (You can get your key by running setup from your SQL database media – make sure you use STD or ENT edition whichever you choose to deploy)
- Accept the License agreement
- Install Reporting Services Only
- Choose a Path, and select Install
- Choose “Configure report server”. You must immediately configure the Report Server.
- Connect to the local server
- Select “Web Service URL” and click Apply
- Select “Database” and click “Change Database”
- Action: Create a new report server database and click Next
- Database Server: Click Test Connection then click Next
- Database: Accept “ReportServer” for default name and click Next
- Credentials: Accept default Service Credentials and click Next
- Summary: Click Next, then Finish when completed.
- Select “Web Portal URL” and click Apply
- Select “Email Settings” and configure your SMTP server in order to be able to use emailed reports in SCOM, and click Apply.
- Now that configuration is done, click Exit
- Validate SSRS is working:
- Open a Web Browser on the server.
- Browse to http://localhost/reports/
- You MUST see a “Home” screen before continuing to install SCOM reporting role.
- Install SCOM Reporting Role on the SSRS SQL server
- Locate the SCOM media. Run Setup.exe. Click Install.
- Select Reporting Server
- Accept or change the default install path and click Next.
- Resolve any issues with prerequisites, and click Next.
- Accept the license and click Next.
- Type in the name of a management server, and click Next.
- Choose the correct local SQL reporting instance and click Next.
- Enter in the DOMAIN\OMREAD account when prompted. You MUST input the same account here that you used for the OM DW Reader account when you installed the first management server. Click Next.
- On the Diagnostic Data screen – click Next.
- Turn Microsoft Updates OFF for SCOM, Next.
- Click Install.
- Close when complete.
- Configure Report extensions
- Open SQL Management Studio on a server where you have this installed, or install it locally.
- Connect to Reporting Services and type in OMSQL1 for the server name.
- Right Click OMSQL1 and choose properties.
- Select Advanced
- Scroll down to “AllowedResourceExtensionsForUpload”
- Add “*.*” to the end of the list of allowed extensions.
You have a fully deployed SCOM Management group at this point.
Open/Reopen the SCOM consoles, ensure you have a Reporting tab now, and within an hour you should see reports populated in the console.
Look for any health issues or alerts, and review the OpsMgr event logs on both management servers for errors or warnings.
Known / Common issues:
1. Management Server installation fails when TLS 1.0 is disabled, and prerequisites for TLS 1.2 are missing.
- On the first management server being installed, the UI will return a failure, and in the OpsMgrSetupWizard.log (found at C:\Users\<username>\AppData\Local\SCOM\LOGS), you see the following:
[09:41:56]: Info: :Info:GetLocalizedAdminGroupName: Administrators Group Name is: BUILTIN\Administrators
[09:42:12]: Error: :PopulateUserRoles: failed : Threw Exception.Type: System.ArgumentException, Exception Error Code: 0x80070057
- On the any additional management servers being installed, this will show up by hanging on “Registering Management Server” and never completing.
- This is caused by having TLS 1.0 disabled on the SCOM management server or SQL server. If TLS 1.2 is enforced or TLS 1.0 disabled, you must FIRST install the software prerequisites for TLS 1.2 for SCOM.
2. When using SSRS 2017 or SSRS 2019, you might see errors on a management server for event ID 31567 with description “Failed to deploy reporting component to the SQL Server Reporting Services server” and “extension is not allowed”. This is apparently because of a new security restriction in later builds of SSRS 2017. The workaround is to open SQL Management Studio, connect to your Reporting Services instance, open the Properties of the instance, Advanced, and add *.* to the list for “AllowedResourceExtensionsForUpload”
3. When using a scoped user profile, you might see a “500 – Internal server error” when using a state view in the Web Console. You also might see an error in state views may for: Incorrect syntax near the keyword ‘CREATE’. This issue was first resolved in SCOM 2019 UR1.
What’s next?
Once you have SCOM up and running, these are some good next steps to consider for getting some use out of it and keep it running smoothly:
1. Configure SCOM Security
- You must add your OMAdmins Global group to allow “Log on as a service” right on Management Servers, in order to push agents. https://kevinholman.com/2019/03/14/security-changes-in-scom-2019-log-on-as-a-service/
- Add your OMAdmins Global Group to the SCOM Administrators User Role. Ensure you, your team, and the SCOM DAS and Action accounts are members of this group FIRST. Then, remove BUILTIN\Administrators from the Operations Manager Administrators – User Role, to secure your SCOM installation.
2. Apply the latest Update Rollup.
- At the time of this blog posting the latest update rollup is UR3. You should always find and apply the most current CUMULATIVE update rollup.
- UR3 for SCOM 2019 – Step by Step – Kevin Holman’s Blog
3. Set SCOM License.
- By default all SCOM installations deploy as “180 Day Evaluation”. You need to apply your license key to make sure your production deployments don’t time-bomb.
- https://kevinholman.com/2017/06/29/dont-forget-to-license-your-scom-2016-deployments/
4. Optimize SQL Server for growth and performance
- Pre-size the OpsDB: When we installed each database, we used the default of 1GB (1000MB). This is not a good setting for steady state as our databases will need to grow larger than that very soon. We need to pre-grow these to allow for enough free space for maintenance operations, and to keep from having lots of auto-growth activities which impact performance during normal operations. A good rule of thumb for most deployments of OpsMgr is to set the OpsDB to 50GB for the data file and 25GB for the transaction log file. This can be smaller for POC’s/LAB’s but generally you never want to have an OpsDB set less than 10GB/5GB. Setting the transaction log to 50% of the DB size for the OpsDB is a good rule of thumb.
- Pre-size the Data Warehouse: You will need to plan for the space you expect to need using the sizing tools available and pre-size this from time to time so that lots of smaller autogrowths do not occur. The sizing helper is available at: DOWNLOAD
- Limit SQL MAX memory reserving memory for the OS.
- Set Power Management plan in OS to “High Performance”
- (Optional) Create a high performance TempDB: (This is already configured by default in SQL 2016 and later) Optimizing TempDB Performance
- (Optional) Optimize MAXDOP: Optimizing MAXDOP
- If you have a SQL Always On scenario – the secondary replicas need a SQL script run on them: https://kevinholman.com/2017/08/27/scom-2016-event-18054-errors-in-the-sql-application-log/
5. Set up SQL maintenance jobs.
- Be proactive. Set up your Database Backups, Transaction Log backups, and your Re-index jobs now.
- https://kevinholman.com/2017/08/03/what-sql-maintenance-should-i-perform-on-my-scom-2016-databases/
6. Configure Data Warehouse Retention.
- This should be done up front, don’t wait for the DW to fill your disks with data you aren’t required to keep.
- https://kevinholman.com/2010/01/05/understanding-and-modifying-data-warehouse-retention-and-grooming/
7. Optimize your management servers registry
8. Enable Agent Proxy as a default setting
- I prefer to simply enable agent proxy for all agents. The BEST way to do this is to enable Agent Proxy as a default setting. That way you will never have to mess with this again:
- https://kevinholman.com/2017/03/10/enable-proxy-as-a-default-setting-in-scom-2016/
9. Configure Administration Settings per your requirements:
- Database Grooming
- Automatic Alert Resolution
- Heartbeat configuration (modify only if required)
- Manual Agent Installs (Reject, Review, or Accept)
10. Backup Unsealed Management packs
- You need to set this up so that in case of a disaster, or an unplanned change, you will have a simple back-out or recovery plan that wont require a brute force restore of your databases. I have seen this save many a customer’s bacon when they had this available, and cause them great pain when it wasn’t.
- https://kevinholman.com/2017/07/07/scom-2012-and-2016-unsealed-mp-backup/
11. Deploy an agent to the SQL DB server.
- This process has not changed from OpsMgr 2012, so you would use the typical mechanism to push or manually install. You can also refer to: SCOM Agent Deployment
- You could also deploy any additional agents at this point.
12. Import management packs.
- https://docs.microsoft.com/en-us/system-center/scom/manage-mp-import-remove-delete?view=sc-om-2019
- Using the console – you can import MP’s using the catalog, or directly importing from disk. I recommend always downloading MP’s and importing from disk. You should keep a MP repository of all MP’s both current and previous, both for disaster recovery and in the case you need to revert to an older MP at any time.
- Import the Base OS and SQL MP’s at a minimum.
13. Configure Notifications:
14. Deploy Unix and Linux Agents
15. Configure Network Monitoring
16. Configure SQL MP RunAs Security:
17. Continue with optional activities from the Quick Reference guide:
18. (Optional) Configure your management group to support APM monitoring.
- https://docs.microsoft.com/en-us/previous-versions/system-center/system-center-2012-r2/hh543994(v=sc.12)
- Import supporting management packs for IIS 8, and 10, and APM Web for IIS 8, and 10.
19. (Optional) Deploy Audit Collection Services
- https://docs.microsoft.com/en-us/system-center/scom/deploy-install-acs?view=sc-om-2019
- Install the audit collector on a management server, and create a database on a SQL server.
- Upload the reports for ACS, my command is: UploadAuditReports.cmd “SQL1” http://SQL1/ReportServer “c:\acs”
- Create and set a filter:
- https://docs.microsoft.com/en-us/previous-versions/system-center/system-center-2012-R2/hh230740(v=sc.12)
- https://kevinholman.com/2017/02/16/how-to-test-your-acs-filter-to-ensure-it-is-valid/
- My initial filter for lab use is: adtadmin /setquery /query:”SELECT * FROM AdtsEvent WHERE NOT (EventId=4768 OR EventId=4769 OR EventId=4624 OR EventId=4634 OR EventId=4672 OR EventId=4776)”
- You will need to grant NETWORK SERVICE full control to the AdtServer registry key to set a filter at the command line: http://social.technet.microsoft.com/Forums/en-US/operationsmanagerreporting/thread/ab22685e-36a1-49a9-b90e-d39ead31901f
20. Learn MP authoring.
- Fragments the fast and easy way with Visual Studio: https://kevinholman.com/2016/06/04/authoring-management-packs-the-fast-and-easy-way-using-visual-studio/
- Download MPAuthor: http://www.silect.com/mp-author/
Will other System Center products need to be updated before I can move to SCOM 2019, or could I potentially update right away?
The recommended upgrade order is posted in the online product documentation.
But could it be, that SCOM 2019 supported agent platform is only Windows Server 2016 and above?
What about Windows Server 2008 – 2012 R2, are they going to be left out for SCOM 2019?
Windows 2012R2 is supported as an agent. We are updating documentation. I am trying to get clear confirmation if Windows Server 2012 (non-R2) is truly unsupported or a documentation oversight. As to Windows Server 2008 and 2008R2, those are not supported with a SCOM 2019 agent. Those OS versions will fall out of Extended Support by Microsoft in Jan 2020, so they were not included in this release.
Hey Kevin,
Server 2008R2 is not supported with a SCOM 2019 agent, but is it supported with the current 2012 MMA connected to a SCOM 2019 management group? If I recall Server 2003 was supported in a SCOM 2012R2 management group with the old agent version, but not on the new (at the time) MMA.
We don’t have any support statements for connecting 2008R2 to a SCOM 2019 management group, with any MMA. So the assumption is, that is something is not clearly defined as supported, then it isnt. This is similar to SCOM 2016 dropping support for 2003. You could still connect 2003 servers to SCOM 2016 using a SCOM 2012R2 agent, but it wasn’t a “supported configuration” according to our documentation. It worked fine. We just didn’t test/support it, because WS 2003 extended support expired, and we generally do not test or develop new products to work with unsupported products. I think the heartache is that WS2008/2008R2 is not expired yet, although it will expire soon, in Jan 2020. Why they dropped WS2012 OS, I don’t have a good answer, and we are pushing them to change on that one. The challenge with all of this, is that the Log Analytics MMA supports WS2008R2 and later, AND supports connecting to SCOM 2012 SP1 UR6 and later.
Hello, I want to update Audit Collection Services for UNIX/Linux, but after removing old version (1807) I can’t install new. Setup is stuck at Prerequisite Check Did Not Pass: Operations Manager is required to be installed prior to the installation of this product. SCOM 2019 is installed and running on this machine, and also Audit collection services for Windows.
Any idea?
Ugh. This is a bug. To work around this – change the reg key at HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup\CurrentVersion TEMPORARILY from “10.19.10050.0” to “7.2.11719.0” Then run setup. This should work around the installer prereq check. Make sure you change this value back after you install this.
Setup went OK. But I still get alert:
Event Description: Loading managed module type in assembly “Microsoft.SystemCenter.CrossPlatform.ACS” with type name “Microsoft.SystemCenter.CrossPlatform.ACS.ACSWriteAction” failed with error “The module assembly “Microsoft.SystemCenter.CrossPlatform.ACS” could not be loaded. The exception was: \nCould not load file or assembly ‘Microsoft.SystemCenter.CrossPlatform.ACS’ or one of its dependencies. The system cannot find the file specified..”.
This may be because the type or assembly could not be found or the type does not have the MonitoringModuleAttribute.
Workflow: Microsoft.ACS.Linux.RHEL.6.Su.Failed
Is this because support for RHEL 6 was removed? This alert has 50 repeats.
Grega
It is not supported, but it does work. I moved was able to somehow setup 2008 Servers in my SCOM 2019 POC environment. It does throw some alerts once it checks in with the 2019 MG, but I dont have that information and the POC environment has since been torn down. We eliminated over 250 2008 Servers in the past 8 months we are now only 2012 R2 and above. I think I will be keeping the legacy environment around for a bit and migrate our 2016 and 2019 servers over to the new and start validating 2012 R2 before I tear that down as well.
Hi Kevin, as usual very good post. Do you know if Microsoft has the intention to fix the xSCOM dsc resource so that we can use it to deploy SCOM 2019 ?
https://www.powershellgallery.com/packages/xSCOM/1.3.3.0
As that has not been updated since 2015, and I have not heard anything about that – I seriously doubt it. That project was open sourced, regardless: https://github.com/PowerShell/xSCOM
You dont need xSCOM DSC resource. You can easily use PackageManagement instead
Hi Kevin,
do you know when Microsoft is going to release the updated Linux/Unix management packs ?
https://www.microsoft.com/en-us/download/details.aspx?id=29696 still only has the 1807 packs and we are eagerly waiting for the SuSE 12 PPC support.
My bad… I should have thought twice before posting 🙂 Of course they are in the installation folder of SCOM 2019
All good now 🙂
Kevin, again an excellent piece of work! Run into an issue with SQL 2017 reporting services. Everything is working fine except http://localhost/reports/ gives a HTTP 500 Internal Server Error after installing. Seems more people run into this issue.
Yes – this is a known issue. It did not repro with the early builds of SSRS 2017, but it evolved in newer versions. The PG is looking into resolving this.
I just installed SCOM 2019 with SQL/SSRS 2017 and ran into this issue.
Is there anything new from the product group on this? Is there an estimated resolution date for it?
thank you!
The most recent SSRS updated build 14.0.600.1274 at https://www.microsoft.com/en-us/download/details.aspx?id=55252 resolves the HTTP 500 error when browsing the /Reports URL.
I also ran into the HTTP 500 error problem on installing the SSRS 2017 and SCOM Reporting on a Windows Server 2019. The quick fix was removing SSRS 2017 and install SSRS 2016, which worked fine.
Later, trying to reproduce this problem I saw that SSRS 2017 is working fine on Windows Server 2016. So it seems to me that the combination SSRS 2017 and Server 2019 is causing the HTTP 500 error.
Discovery Wizard SCOM 1902 show strange.. Now showing entire tabs (Computer select etc…)
We face an issue that once install SCOM(2019) web service in the same server with the SQL(2016) service, the web service can’t access properly. Not sure is SCOM, SQL or IIS issue, any idea to fix ? Thanks.
I never install IIS and SQL SSRS on the same server – this is likely your issue. I install the SCOM web console role on management servers. However, if you want to do this, see: https://docs.microsoft.com/en-us/sql/reporting-services/install-windows/install-reporting-and-internet-information-services-side-by-side?view=sql-server-2017
Hi Kevin,
If I am upgrading the SCOM management servers from 1807 to 2019 in a distributed environment, do I have to repeat this pre-upgrade task on each management server or just on the first MS? “7.Stop the Microsoft Monitoring Agent, System Center Data Access Service, System Center Configuration Management, and Microsoft Monitoring Agent services on all management servers except the one being upgraded.” Any plans on posting a guide for the upgrade process? Thanks!
So, to prepare for upgrading our 1807 deployment to SCOM 2019, I had our DBAs restore the OpMgr and DW DBs from our LAB MG to a new SQL 2017 server (from current SQL 2014). The DBA migrated all permissions as well, and afaik, they are identical and meet all documented requirements. After following all the MS steps to edit the registry keys and configservice.config file on each MS, and editing the listed tables on the DB server for both the OpsMgr and DW DBs, SCOM appears to be running, all MS are talking to SQL, with a fairly robust amount of data being exchanged for our LAB MG, but I can’t launch a console on any MS. Data Access Service fails, with Application Log Errors 1000 and 1026, System Log Error 7034 (OMSDK keeps restarting and failing) , and OpsManager Errors 26340 and 26380. If I restore the previous registry keys values and old configservice.config values, Management servers have no problem talking to the old DB. Any ideas? I’ve Googled for two days and found nothing, even looking at SCSM stuff.
This is typically a rights or TLS issue. Is the new SQL 2017 server using same ports, and configured the same?
Any other ideas for solution or diagnostics? Thanks.
Two of my DBAs swear up and down that logins, perms, ports (default on both old and new servers), firewall, etc. are all configured the same. And the management servers and SQL are talking. Traffic on all of them, and OpsMgr log files show that agents are available. Just no console and the DAS keeps failing, restarting, failing, etc. I notice that in Microsoft’s “How to configure Operations Manager to communicate with SQL Server” page, the database edits specify using (computer\instance,portNumber). On the old server, only SQL server name is used. Since we’re using default ports, my DBAs don’t think this should be an issue, but did this maybe change in SQL 2017? Grasping at straws here (at least it’s in the lab MG). Thanks.
Hi Kevin,Is Windows 2012 R2 supported (will still work) as a gateway role on SCOM 2019? I read the Software requirement for SCOM 2019 and it only covers from win2016. (https://docs.microsoft.com/en-us/system-center/scom/system-requirements?view=sc-om-2019).
Thanks,
Nirmal
For SCOM server roles (such as a GW) only what is documented is supported.
Hi, Kevin if I try to discover a Windows computer (Advanced discovery, Browse for) it doesn’t function. SCOM tries to discover it but it never ends. And it doesn’t matter if I use the Action Account or the other account with the admin rights. Any idea?
Thanks,
Casimir
I cover this here: https://kevinholman.com/2019/03/14/security-changes-in-scom-2019-log-on-as-a-service/
Hi Kevin, one thing more: the discover never ends and I see the info that the Discovery is the SQL Broker necessary.
I was wondering how much SCOM will complain if I do not install the reporting role ? We never really used the reporting in the past and I’m ok if we do not have this piece. However, I suspect I will have tons of msg in the logs about it.
None at all. It is completely optional. The DW is mandatory, and created during setup. However, there is zero requirement for a reporting role if you dont want it.
Note: if you upgrade the management servers to UR1 than you will have problems installing the SCOM Reporting server. I worked with MS tech support for several days and it couldn’t be installed. I ended up having to re-provision my new SCOM 2019 servers and re-install everything. then install UR1.
Hi Karen – I have addressed this and provided a simple solution in my UR1 and UR2 posts.
Hi Kevin,
BTW, you are missing that .net 3.5 framework is required on the Web Console.
Hi Kevin,
I’ve run the 2019 upgrade on two 1807 management servers so far and in both cases the old and new versions of SCOM are shown in appwiz.cpl . In “operations manager products” the consoles on the management servers are detected as 10.19.10050.0 and the management servers are still detected as 7.3.13142.0 – is this expected/a known issue or is it something specific to my environment do you think?
Cheers
Steve
Morning, could you please verify if the URL used to download mgmt packs/updates from within the SCOM console is the same for SCOM 2012R2, 2016 and 2019?
https://www.microsoft.com/mpdownload/ManagementPackCatalogWebService.asmx
ports 80, 443
Our environment has very tight firewall rule controls so I need to get this correct.
Thank you,
Tony
I cannot. The reason is – I do not recommend every using that for downloading MP’s. I consider that a worst practice, as I recommend downloading each MP, extracting it – saving it into a repository, testing it in your lab/test environments, then migrating it to production. Sucking in MP’s from the web breaks that change control process, and removes your ability for a proper disaster recovery and rollback capability, in my opinion. I’m sure someone knows the answer to your question, however, but it isnt something I use with my customers.
Afternoon, I understand your position and generally agree. I want to enable it for the feature “show updates for currently installed management packs” since it also provides version info. I do follow the process you recommend, I think it is the safest route to go. We have a good change management process that requires us to document that updates have been tested and properly configured in “lower tiers” prior to migrating/moving on.
Thanks as always, for your time and assistance.
Regards,
Tony
As far as I know – the link you posted is the one we use for MP downloads and updates.
Hey Kevin,
This is a great guide!
After installing SSRS 2017, you mention “Browse to http://localhost/reports/“. Did you mean “http://localhost/reportserver”?
I’m asking because “http://localhost/reports/” does not appear to work. It gives me “HTTP 404 Not Found”.
Navigating to “http://localhost/reportserver” shows “servername/ReportServer – /, Microsoft SQL Server Reporting Services Version 14.0.600.1109”
I’m staging a POC of SCOM 2019 on Server 2016.
Ignore this.. I missed a step and am able to navigate to the URL “http://localhost/reports/”.
Would like to mention that the web console is not being installed as it’s not required.
After running the SCOM Reporting role on the SQL server, it fails and rolls back.
Kevin-
Thanks so much for this!
I’m getting the following error in the Application log in Event Viewer on some of my my test SCCM servers (source is Perflib): “Windows cannot open the 64-bit extensible counter DLL MOMConnector in a 32-bit environment.” Any ideas? It’s puzzling as they are 64 bit scom agents and 64 bit OS systems..
Is there a current .inf file for requesting certificates for SCOM communications?
I’m finding older ones, but I’m not sure if these are the best choice for settings, for TLS 1.2, etc… for best security.
Thanks for a great guide Kevin
I have installed SCOM2019 with a new management group and shall now remove the old one from Active Directory. When but I receive this error executing ” .\MomADAdmin.exe -d SKANDIKON SKANDIKON” with my domain admin user.
MomADAdmin failed to delete the container for SKANDIKON with the following exception:
Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Can I delete the old management group under OU OperationsManager manually from Active Directory?
Yes
Hi Kevin,
I have upgraded 2 Win2016 MS from v1807 to 2019 and everything seems to be reporting the correctly upgraded version except the database and data warehouse (7.3.13142.0). The SQL server is SQL 2016 Std. on Win2016 as well, is this normal and if not what might I have done to not have this update correctly?
Thanks for your time
Hi Kevin,
thank you so much for your blog. It is very useful!!
We have installed everything and we are suffering the “Known Issues number 2”: We can browse reports ever but not Reports. We have updated the SQL from 17 to 17 with CU14. But the problem persists.
Do you know which version we need to solve the issue? Or do we need to change the SQL version?
Thank you again!
The SQL team should be releasing an update for SSRS which will address this soon.
The most recent SSRS updated build 14.0.600.1274 at https://www.microsoft.com/en-us/download/details.aspx?id=55252 resolves the HTTP 500 error when browsing the /Reports URL.
Thank you Kevin. it works properly!
HI Kevin,
We were able to install the 2019 SCOM Management Server using a remote instance of SQL.. When trying to install Reporting Services I am not able to select a remote instance on the “SQL Server instance for Reporting Services”. It’s grayed out and I am not able to type in the box.. We went back to the “Select Features to install” page and noticed at the bottom of the page under Requires: “This Feature requires a local SQL Reporting Services instance to be installed. Refer to the Operations Manager Supported Configuration document for the full list of system requirements”. Is there a way to install 2019 SCOM reporting services with a remote SQL instance or does it truly have to have SQL installed locally for reporting services?
SCOM Reporting Role must be installed on a local SQL Reporting Server. It has always been this way. The SCOM Reporting role is very minimal. Most customers just install this role and SSRS on their Data Warehouse server, or in larger environments, they install SSRS and the SCOM Reporting role on a dedicated server.
Some more back ground… We have a remote SQL server. I have installed a separate instance of SQL and reporting services on the remote instances specifically for reporting services. The ReportServer and ReportServerTempDB exist.. I used the Report Server Configuration manager on the Scom server and have connected to the database and web service url and web portal url by modifying the url’s and pointing to the the sql server and SSRS instance. While I can’t find anywhere that specifically states that says the SSRS instance must be local to the SCOM server it seems this needs to be the case. I’m looking for a confirmation.
Thanks
You replied as I was typing up my follow up note. This is my first go around with SCOM.. I think I get it now..
Thanks
Hi Kevin,
Just wanted to check on the Power Settings. Is High Performance Power only needed for the SQL Server, or does this also apply to all Management Servers?
In a past company, which was very large…8 management groups… we were advised about this during a PS case, and at that time it was implied that it was all MS’s too, which we did, and we did see improvements. However, whilst implementing this elsewhere it was queried and I see that on this article, it is only mentioned within the SQL section.
So, I guess I am asking is it really necessary/safe to apply this to the Management Servers, or could there be another issue lurking.
Thanks
My understanding is that this setting has no effect on VM’s at the guest level, so it really doesnt matter, since management servers are always VM’s in my experience. It matters much more when SQL servers are deployed on physical hardware.
Hi Kevin,
How about upgrading the other SCOM management servers in a distributed environment. The first one did the upgrade with success from 1807, but the other three management servers will not be upgraded. How can I force that process? By the way, that are 2016 Core servers. Setup program is not supported on a Server Core installation.
What does “will not be upgraded” mean? Does it fail? If so – look in the log to see where it s failing.
On server core – you need to use the command line. I have never tested using server core and doing a SCOM version upgrade.
Thx Kevin for your quick response.
I’ve got also OpsMgr Management Configuration events:29112 ‘Service failed to execute bootstrap work item ‘ . So a bootstrap upgrade procedure will start at the other management servers, but fail with
‘Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Service binaries (cscmdbops.dll) version of 7.3.13142.0 is lower than minimum required Cmdb support version of 10.19.10050.0 recorded in Cmdb. Service binaries must be updated to version no lower than version of the Cmdb support objects. Alternatively Cmdb support objects may be rebuilt to match the version of binaries at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.StoreInitializationOperataion.StoreVersionObtained(Object sender, DataAccessOperationCompletedEventArgs args)’
Our Management Configuration Service runs with the same Log On account as the Data Access Service
(SOLVED)
This command will do the job:
Start-Process -FilePath “C:\Temp\SCOM2019\setup.exe” -ArgumentList ‘/Upgrade /InstallPath:”D:\Program Files\Microsoft System Center\Operations Manager” /components:OMServer /ManagementGroupName:XXXX /SqlServerInstance:XXXX /DatabaseName:OperationsManager /DWSqlServerInstance:XXXX /DWDatabaseName:OperationsManagerDW /ActionAccountUser:Domain\XXXX /ActionAccountPassword:******* /DASAccountUser:Domain\XXXXX /DASAccountPassword:****** /EnableErrorReporting:Never /SendCEIPReports:0 /UseMicrosoftUpdate:0 /AcceptEndUserLicenseAgreement:1 /silent’
Hello Kevin.
Until SCOM 1807 Solaris 11 x86 was supported (https://docs.microsoft.com/en-us/system-center/scom/plan-supported-crossplat-os-1807?view=sc-om-1807)
On the SCOM 2019 page I only see Solaris 11 SPARC Version, not x86 anymore. Do you know if this is an error or if it is correct (“by Design”)?
https://docs.microsoft.com/en-us/system-center/scom/plan-supported-crossplat-os?view=sc-om-2019
Indeed, if I try to add a Solaris 11 x86 System that worked with SCOM 2012R2, in SCOM 2019 I get a Message “not supported”.
I’m just wondering why?
Great question. I don’t know. My guess would be lack of customer adoption of Solaris 11 on x86…. but I’m not sure the criteria used by the product group to choose which UNIX/Linux versions got support moving forward and which didn’t.
Hi,
in the MP guide both, x86 and SPARC, are listed as supported. Still, the kit for Solaris 11 i386 is missing…
Anyone found anything on that?
Cheers,
Patrick
Hi Kevin,
Thanks for a great article.
I have deployed many OPS managers from 2007 to 2016, but this is the first 2019, and no matter how many times i have re-installed this installation ( i thing i have installed it 6-7 times now – no matter what installation guide i follow ore have installede like i used to do) and for some reason i keep getting the same error and can’t figure out why.
Maby you know why our can guide me in the right direction?
Data Warehouse failed to deploy reports for a management pack to SQL Reporting Services Server
Data Warehouse failed to deploy reports for a management pack to SQL Reporting Services Server. Failed to deploy reporting component to the SQL Server Reporting Services server. The operation will be retried.
Exception ‘DeploymentException’: Failed to deploy reports for management pack with version dependent id ’84a5e876-f9eb-a3b1-e566-b7f7c2fe9dcf’. Uploading or saving files with .Settings extension is not allowed. Contact your administrator if you have any questions. —> Microsoft.ReportingServices.Diagnostics.Utilities.ResourceFileFormatNotAllowedException: Uploading or saving files with .Settings extension is not allowed. Contact your administrator if you have any questions.
One or more workflows were affected by this.
Workflow name: Microsoft.SystemCenter.DataWarehouse.Deployment.Report
Instance name: Data Warehouse Synchronization Service
Instance ID: {2D8188D5-F0F4-6A12-FD7F-0FCDD6E6B445}
Management group: XXXXXXXXXXXX
That’s #3 in the KNOWN ISSUES section of this guide.
🙂
Thanks a million! 🙂
Don’t know how I could miss that one…
In fairness – I only added it a few days ago. 🙂
Hi Kevin
as always an excellent article ?
is there a description how to install a gateway on a core server?
Thanks
Bruno
Finally I found, in short:
certutil -importpfx certificate.pfx
Microsoft.EnterpriseManagement.GatewayApprovalTool /managementServerName=MS-FQN /GatewayName=GW-FQN /Action=Create
msiexec.exe /i MOMGateway.msi
momcertimport64.exe
Hi Kevin,
SCOM 2019 Linux discovery is failing with below error, could you please advise any workaround for this. The Winrm service is running.
Unexpected DiscoveryResult.ErrorData type. Please file bug report.
ErrorData: System.ArgumentNullException
Value cannot be null.
Parameter name: lhs
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs, TimeSpan timeout, WorkflowInstanceExtensionManager extensions)
at Microsoft.SystemCenter.CrossPlatform.ClientActions.DefaultDiscovery.InvokeWorkflow(IManagedObject managementActionPoint, DiscoveryTargetEndpoint criteria, IInstallableAgents installableAgents)
When we run Winrm below error is comming
WSManFault
Message = The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: “winrm quickconfig”.
Error number: -2144108526 0x80338012
The client cannot connect to the destination specified in the request. Verify that the service on the destination is running and is accepting requests. Consult the logs and documentation for the WS-Management service running on the destination, most commonly IIS or WinRM. If the destination is the WinRM service, run the following command on the destination to analyze and configure the WinRM service: “winrm quickconfig”.
Regards,
Kiran Kumar Reddy
I solved this problem by deleting old SCOM agents from the directory on all SCOM Management Servers and SCOM Gateways. When deleted I only needed 4 files “scx-1.6.4-7…..” for my environment.
C:\Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits
And then discovery worked instantly.
SCOM 2019 / Windows Server 2019 / SQL Server 2017 Std / SSRS 2017 on DW server. Installing Scom reporting service role on Datawarehouse server fails always. Installation fails when Operations Manager Setup is configuring security extensions. After that SSRS Reporting services http://localhost/Reports is also broken.
Is this “known issue” or should I open ticket for MS for this case?
Not a known issue. Reporting should install fine if SSRS is properly installed and working prior to the SCOM reporting install.
Hi Jukka-Pekka Grohn,
could you fix the error? Seams i run in the same. Everything i try runs in the same error.
Would nice to hear from you.
Thank you
Hi Everyone,
This is a Permissions issue with the account being used to do the Reporting install.
I recommend validating the account being used has Admin rights to the database and all SCOM Servers.
As a side note you might even ensure the SCOM Read account has SysAdmin rights on the database just for the install.
Has anyone had issues with the webconsole in 2019, i have tried everything i can think of but its not loading correctly i don’t get the side navigation and get the central dashboard squeezed in to where the navigation should be and the rest of the screen is just blue.
Afternoon, I am at a total loss. Followed the document to a tee and I get the 0x80070057 error, everything fails. did line for line of the instructions. sql server 2016,sp2 windows server 2016, 2 mgmt servers windows server 2016, one for mgmt server and console, other for web and reporting. did not make it past installing for the first mgmt server!! so frustrating with something that should be so simple! all prereqs met, passed all the screens to proceed. do I not need to do the tls 1.2 since windows server is 1.2 compliant? do I need tls 1.0 also????? thanks for your time and assistance. TS
If TLS 1.0 is disabled, you just need to apply the TLS prerequisites FIRST, before attempting an install. SCOM 2019 will install on a TLS 1.2 enforced environment, but not without the TLS 1.2 prerequisites first.
Hi Kevin,
I have installed SCOM 2019, running is OK. I just want to allow another user (Windows guy) to monitor or administrator only windows devices. In this case I have allowed a separate user under ‘Securtiy > User Roles’. Whatever tutorial I followed, it seems that the user find access to everything as the SCOM administrator find !! have logged in web console each time. Its been 3 days, no single clue I have been able to identify !
In the ‘Group Scope’ and ‘Dashboards and Views’ I selected only Windows items. Very much appreciable your kind response in this regards.
By default, any local administrator of a SCOM server OS is all SCOM admin by inheritance. This is the first thing I tell people to change in my guides.
Hi Kevin, Thank you a lot.
Another question.
is it possible to allow a specific Network departmemt’s user to install agent on their rest of network devices? Where the user can see only network devices only.
Allowing the network user in User Roles with ‘Profile: Advance Operator’ can’t see any option/wizard to install agent on Linux Machine!! Or if you tell how to do this would be much helpful.
Is this has any mandatory option to login from desktop console with that user instead of Web Console?
Appreciate your tremendous help/answer.
Hi Kevin. I downloaded SCOM 2019 from VLSC and it also offered me SQL Server 2019 Standard Edition under the same section that the SCOM 2019 key was (as was SQL Server 2017 Standard Edition). I thought I’d test SQL Server 2019 considering it was offered in the same section, however it fails to install when it tries to import the Management Packs.
Totally appreciate that SQL Server 2019 doesn’t appear to be supported yet, but thought it was worth a crack considering it was offered to me to download in the same section; under System Center 2019 – Operations Manager Server.
Do you know if this is definitely not supported yet? Perhaps they put it there as it will be later and maybe only for an upgrade from SQL Server 2017 from an existing install?
Thanks
SQL 2019 is not supported for SCOM 2019. SQL 2019 just shipped, 11/4/2019. It was not available at the time SCOM was shipped. SCOM 2019 may add support for SQL 2019, but that will be some time in the future.
My System Center Datacenter comes with SQL 2019 — are you saying that MS is ship a product with incompatible components?
The System Center 2019 Datacenter licensing comes with usage rights for SQL standard edition, that much is true. However, I am not aware that we state SQL 2019 anywhere. We do state that it includes SQL server standard edition runtime to support System Center deployments. You still must ensure you are using a supported version of SQL server for the System Center product.
Thanks Kevin, thought so. Just odd that it was offered with the VL version of SCOM 2019 on the VLSC site.
“You also will see the same bug that was in SCOM 2016 UR6 where state views may throw an error: Incorrect syntax near the keyword ‘CREATE’. This bug was fixed in SCOM 2016 UR7 and is scheduled to be resolved for SCOM 2019 in SCOM 2019 UR1.”
There is a fix for this: https://support.microsoft.com/en-my/help/4506518/system-center-operations-manager-hotfix-for-scoped-group-users
Hi Kevin,
I’ve been breaking my head about installation issue of SCOM for a day now, everytime I disabled the firewall on my SQL server it would continue to next step, when it was disabled it did not continue although I was 100% sure the appropiate firewall rules were setup (1433 was working and accessible).
Perhaps a good side-note for your readers and perhaps even worthwhile putting it in your steps. For SCOM to access the SQL server you not only need the SQL port open but also “Windows Management” also known as “WMI-In” in the firewall. If you enable this together with the SQL port it will work.
Thanks for your how to quickstart, I have it bookmarked to make my life easier as probably a lot of people do 🙂
Hi Kevin, FYI step # 11 above is pointing to a broken link (404) https://docs.microsoft.com/en-us/system-center/om/manage/managing-discovery-and-agents?redirectedfrom=MSDN
Trying to install a new group.
I’m not getting the step: “On the specify an installation screen – choose to create the first management server in a new management group.”
It immediately asks for a group to connect to.
Kevin, i have a question about SCOM 2019 and it’s SQL Server Collation Requirements. I’m running up a test 2019 system and what it to reside on a SQL 2016 Always ON cluster. This SQL 2016 Always ON cluster has been configured with a different SQL Server Collation that the one required for SCOM 2019. As a result my installation fails at the database check – saying the SQL Server Collation is incorrect. My question is – does the SQL Server Collation need to be system wide for SCOM to function correctly (as i know it can be set per DB). Could i for example install my test SCOM db onto a different instance of SQL 2016 (that has the correct SQL Server Collation) and then move my SCOM db’s to the SQL 2016 Always ON cluster, that has the wrong SQL Server Collation, the SQL Server Collation can be per database, so my SCOM db’s should have the correct collation. but will the system collation settings of the SQL 2016 Always ON cluster break my SCOM db’s / install. thanks in advance and sorry for the longwinded explanation. karl
No – that would not be supported, because we have issues when the tempDB’s are a different collation. SCOM is not a simple database application, we tightly integrate with the SQL instances and leverage multiple services, like CLR, fulltext search, broker, custom messaging in the master database, and TempDB. The SCOM databases and the SQL instances themselves must use the supported collation.
Thanks Kevin, just what i thought. Now i have to break it to my SQL dba’s!
Hey, thanks for all your guides.
We ran into an issue with installing the Reporting Server.
The services are correct installed and i can run the homescreen.
When i install the report server feature we stuck at specify a management server with:
Unable to connect to the Data Access service for this management server. Ensure the Data Sccess service is running and the service, the management groupm and setup are all the same version.
telnet scom dw DB –> scom mgmt server 5723/5724 is open.
Data access service is running
Anyone facing the same error?
Thanks and kind regards,
Alex
Hello Kevin,
Today I have install SCOM 2019 with SQL DB 2019. While scom agent installation my agent is not coming under “agent Managed” since I have click “Automatically approve new manually installed agent”.
1. I have tried manual agent installation ..no luck
2. I have tried push agent installation .no luck (stuck in pending management – only “Reject” option is enable
SQL Version : Microsoft SQL Server 2019 (RTM-CU8) (KB4577194) – 15.0.4073.23 (X64) Sep 23 2020 16:03:08 Copyright (C) 2019 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2019 Standard 10.0
I had received below error but I have resolved this issue with your post on TLS 1.2 blog:
OleDb Module encountered a failure 0x80004005 during execution and will post it as output data item. Unspecified error
: [DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security error.
Workflow name: Microsoft.SystemCenter.SqlBrokerAvailabilityMonitorForPool
Instance name: All Management Servers Resource Pool
Instance ID: {4932D8F0-C8E2-2F4B-288E-3ED98A340B9F}
Requesting your help…
Hi i have this problem on install :
FW disabled, using only omaa account is in administrator and sysaadmin on SQL.
[00:05:38]: Always: :ImportManagementPack: Loading management pack D:\NEW\Install\SCOM\Setup\AMD64\..\..\ManagementPacks\System.Library.mp. 00:05:38
[00:05:43]: Error: :ImportManagementPack: Error: Unable to load management pack D:\NEW\Install\SCOM\Setup\AMD64\..\..\ManagementPacks\System.Library.mp
[00:05:43]: Error: :: Database error. MPInfra_p_ManagementPackInstall failed with exception:
Database error. MPInfra_p_ManagementPackInstall failed with exception:
Maximum stored procedure, function, trigger, or view nesting level exceeded (limit 32).
[00:05:43]: Error: :ImportManagementPack: Unknown Error. System.ArgumentException : The requested management pack is not valid. See inner exception for details.
Parameter name: managementPack
[00:05:43]: Always: :FirstManagementServer: Failed to load MP D:\NEW\Install\SCOM\Setup\AMD64\..\..\ManagementPacks\System.Library.mp. We will retry.
[00:05:43]: Always: :ImportManagementPack: Loading management pack D:\NEW\Install\SCOM\Setup\AMD64\..\..\ManagementPacks\System.Library.mp. 00:05:43
OK this is problem with SQL 2019 need min CU8
Hi Kevin,
Thanks for this and all that you post, invaluable.
I’m fairly new to SCOM and have been tasked with doing a side-by-side migration/upgrade from SCOM 2012R2 to 2019.
The plan being to have both environments stood up and slowly migrate existing gateways and agents by multi-homing them so they are reporting to both. Once I’ve seen them reporting ok to the new env for a while disconnect them from the old environment.
My question, in order to accomplish this, the new environment, new servers that I will install SCOM 2019 fresh on, the Management group name should be new and different from the existing one correct?
Thanks and kindest regards
Rich
Yes absolutely! The MG name must be different or you cannot multi-home. MG name is not relevant – so just make it short, generic, easy to type, and unique. I like things like MON1, MON2, MON3 etc. Or DEV, PROD, PROD1, PROD2, etc. People typing out these long Management Group names like there is some kind of naming standard to apply, are ridiculous. KISS.
Hi Kevin,
Every time I try and install SCOM2019 with SQL19 I get operational database configuration failed any help would be great first time I installed SCOM Very new to it all.
Thanks
Russell
Did you first apply CU8 or later to all your SQL 2019 servers being used for SCOM?
I didnt but when i did it worked thanks for the Reply Kevin 🙂
I created this SCOM 2019 AutomatedLab to deploy an Hyper-V SCOM Environment from scratch, with one script.
Check it out if you are looking to get a lab set up quickly: https://github.com/v-bldrum/SCOM-Scripts-and-SQL/tree/master/AutomatedLab
Hi Kevin,
Two more questions although they can be very naíf…
1) Are there any restrictions on the name of SCOM DBs? Our db Admin have rules for naming and I’m sure if SCOM supports anything other than OperationsManager or OperatinsManagerDW.
2) Are there any restrictions for creating DBs in shared SQL instances? For now, the installation is in a non-productive environment.
Thanks once again for your time and support.
No restrictions on the DB names that I know of.
We do not mandate dedicated instances. But keep in mind:
1. The SCOM installer needs Sysadmin and Local Admin rights to get through the install.
2. The DB’s need to be created by the installer Setup routine.
3. We modify permissions, master DB, and MSDB in the instance.
4. All our sizing assumes a dedicated instance. You might see performance issues in a shared instance depending on the size of your SCOM deployment.
For Linux monitoring, is domain account required or user account having access on Linux machines as well, will do?
Getting this error while inux installation with a user account that has access on Linux machine. Please assist how to resolve.
Unexpected DiscoveryResult.ErrorData type. Please file bug report.
ErrorData: Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.TaskInvocationException
Task invocation failed with error code -2130771961. Error message was: Creation of module with CLSID “{}” failed with error “Illegal operation attempted on a registry key that has been marked for deletion.” in rule “Microsoft.Unix.WSMan.Discovery.Task” running for instance “UNIX/LINUX monitoring Resource Pool” with id:”{}” in management group “”.
at System.Activities.WorkflowApplication.Invoke(Activity activity, IDictionary`2 inputs, WorkflowInstanceExtensionManager extensions, TimeSpan timeout)
at System.Activities.WorkflowInvoker.Invoke(Activity workflow, IDictionary`2 inputs,
Once you apply UR1, you cannot install, or re-install SCOM reporting Role.
You will see an error in the setup UI (when you supply a management server name) that states “Unable to connect to the Data Access service for this management server. Ensure the Data Access service is running and that the service, the management group, and setup are all the same version.”
Apply the following workaround to install/reinstall SCOM Reporting role:
QUERY the OPERATIONSMANAGER database, and record the VERSION number that is returned. You will need this value later. You need to change the PrincipalName to your SCOM Management server that you point the reporting install to.
— 10.19.10050.0 – 2019 RTM
— 10.19.10311.0 – 2019 UR1
— 10.19.10349.0 – 2019 UR1 with post UR1 Hotfix
USE OperationsManager
SELECT PrincipalName, Version FROM MTV_HealthService
WHERE IsManagementServer = 1 AND PrincipalName = ‘OMMS1.opsmgr.net’
UPDATE the VERSION entry in the OpsDB to match the RTM version number which is 10.19.10050.0 just for this management server.
UPDATE MTV_HealthService
SET Version = ‘10.19.10050.0’ — 2019 RTM
WHERE PrincipalName = ‘OMMS1.opsmgr.net’
Install SCOM 2019 Reporting, and choose this same Management Server. Reporting install will work now.
REVERT the VERSION entry in the OpsDB to match the original value you recorded
UPDATE MTV_HealthService
SET Version = ‘10.19.10311.0’ — 2019 UR1
WHERE PrincipalName = ‘OMMS1.opsmgr.net’
Hello Kevin,
We have installed SCOM 2016 in our environment and have 4 Management servers (All VMs) and two separate database servers (both high end physical servers with RAID 1+0 configured) each for OpSDB and Data Warehouse. We need to configure AEM for roughly 1,00,000 clients with 25000 clients reporting to each management server. We have enabled created a group policy in AD with object level targeting where each Server’s entry is directed to corresponding SCOM AEM Workstations AD group. Currently, we are managing around 20K clients. We sometimes face issues of high memory on any of the Management Server randomly. Going forward we will add more clients for monitoring but before that i have some queries.
1. What is the right/any alternate method of monitoring these much clients on multiple management servers using group policy ?
2. Is Virtualization (only Management Servers) supported for monitoring 1,00,000 AEM clients ?
3. How to find out clients reporting to which server from SCOM end or any other tool.? (Client doesn’t show up where they are reporting to in console)
Is this SCOM management group dedicated to AEM only, or is it also managing Windows Agents?
I do not recommend mixing AEM monitoring with server monitoring at this kind of scale (which your AEM numbers are our maximum)
Yes Kevin. This setup is purely dedicated to AEM only. What are your thoughts on the 3 points I have mentioned above as I can see there is very less information is available on internet for SCOM AEM.
thanks for the information, a question because when updating the packages and restarting it no longer allows the connection to the scom indicating that the port is busy and that the data access is stopped
Hi Kevin,
Thanks for having this platform. I am having issues deploying agents to Linux (RHEL 7 and 8). Push from OpsMan fails with:
Agent verification failed. Error detail: The server certificate on the destination computer (lmgllvtinf5001.zzzz.com:1270) has the following errors:
Encountered an internal error in the SSL library.
It is possible that:
1. The destination certificate is signed by another certificate authority not trusted by the management server.
2. The destination has an invalid certificate, e.g., its common name (CN) does not match the fully qualified domain name (FQDN) used for the connection. The FQDN used for the connection is: lmgllvtinf5001.zzz.com.
3. The servers in the resource pool have not been configured to trust certificates signed by other servers in the pool.
The server certificate on the destination computer (lmgllvtinf5001.zzzzzzz.com:1270) has the following errors:
Encountered an internal error in the SSL library.
It is possible that:
1. The destination certificate is signed by another certificate authority not trusted by the management server.
2. The destination has an invalid certificate, e.g., its common name (CN) does not match the fully qualified domain name (FQDN) used for the connection. The FQDN used for the connection is: lmgllvtinf5001.hcplab.local.
3. The servers in the resource pool have not been configured to trust certificates signed by other servers in the pool.
I am running SCOM 2019 with a Linux Resource Pool with 2 MGMT servers and followed cert exports/imports per your KB’s. Seems to be cipher related but cannot be certain. I get the same failure running WinRM Enumerate to the Linux Server in question:
WSManFault
Message = The server certificate on the destination computer (lmgllvtinf5001.zzzzzz.com:1270) has the following errors:
Encountered an internal error in the SSL library.
Error number: -2147012721 0x80072F8F
Here is the output from the Linux server with verifying the cert:
sysops1@lmgllvtinf5001:/etc/opt/omi/ssl # openssl s_client -connect localhost:1270
CONNECTED(00000003)
depth=0 CN = lmgllvtinf5001.hcplab.local, CN = lmgllvtinf5001.hcplab.local
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = lmgllvtinf5001.hcplab.local, CN = lmgllvtinf5001.hcplab.local
verify error:num=21:unable to verify the first certificate
verify return:1
—
Certificate chain
0 s:/CN=lmgllvtinf5001.hcplab.local/CN=lmgllvtinf5001.hcplab.local
i:/CN=SCX-Certificate/title=SCX55a8126f-c88a-4701-9754-8625324426ff/DC=LMGLWVCMGT4002
—
Server certificate
—–BEGIN CERTIFICATE—–
MIIDRTCCAi0CAQEwDQYJKoZIhvcNAQELBQAwbDEYMBYGA1UEAxMPU0NYLUNlcnRp
ZmljYXRlMTAwLgYDVQQMEydTQ1g1NWE4MTI2Zi1jODhhLTQ3MDEtOTc1NC04NjI1
MzI0NDI2ZmYxHjAcBgoJkiaJk/IsZAEZFg5MTUdMV1ZDTUdUNDAwMjAeFw0yMTAy
MTIwMDUyMjJaFw0zMjAyMTIwMTExNDFaMEwxJDAiBgNVBAMMG2xtZ2xsdnRpbmY1
MDAxLmhjcGxhYi5sb2NhbDEkMCIGA1UEAwwbbG1nbGx2dGluZjUwMDEuaGNwbGFi
LmxvY2FsMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAztH94mfSyPTH
CNTWtD/6xTsh3QP/VwFj+Tk56UjQLfSA7i8rmf99vNDEDEqYu37dFFb4iSrOwpgP
XCF2sVdgbEBGDrn5Ac+RImgY9XQDRB9RULVS5qDPpmjV3G6Zx6G6yM9WhP9lx3vH
ee59r9eYbj8QifrntpsDMWVr+d2XjSD7mVLzAXB3Jh3yO+lpVyy2fvlUYca2q+BD
MtLeCamS8OJMOy28oBLDwAfMMwkUo+FoOQ1nj9IBVPWlNmOGYubhDcx6Ih/IYFQs
YgXTmkqcOlGLzamNjwWSCHzMcPAbUZlhWHNPZdz1zBRu2mi1IGyMHvj4t0ALTU4C
QsuoR+jcZQIDAQABoxcwFTATBgNVHSUEDDAKBggrBgEFBQcDATANBgkqhkiG9w0B
AQsFAAOCAQEADXwPzPKySe1KvCaFhEwuO7eqJHJnk+4E3pgZbNX5imjo3ldmpHkc
vJ1yn/Aix/chaD3B/fTe07KWrjsGmlyzDruCLMCn3SzAiP++2fjHtypPdd/APGDK
uJTjZxfcUaPs4Oi8C0ZXgiSLUDCgWIkJbIVXL45u7yE6xiQikOXZBzbLbirjhwzr
tPVpVvr/50FWWVYzXmRTv29vF1Yi/WVokh1W3UnKsLUwkN3odyQJtMUqMWleNQYW
tPK1v7qHWiiPZTQ/lMxSp09DIL1J9k2hDJ01dU6bTq8oqUDtkGKLJCq7fvmWVrMd
oepylTL07wwMeMm94DH6yFM3tA9soIrNgg==
—–END CERTIFICATE—–
subject=/CN=lmgllvtinf5001.hcplab.local/CN=lmgllvtinf5001.hcplab.local
issuer=/CN=SCX-Certificate/title=SCX55a8126f-c88a-4701-9754-8625324426ff/DC=LMGLWVCMGT4002
—
No client certificate CA names sent
—
SSL handshake has read 1154 bytes and written 607 bytes
—
New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-GCM-SHA384
Session-ID: 6FB3880342F3670DE7970F27FBA5C31B3C10DC1C834922C0DB6863B0A9409DB7
Session-ID-ctx:
Master-Key: 27EB40E69E555D01C7C7B7B3B392A2FA9C002F802554213DB5ADA7B7C964AA87B289663E8052B331C5215F7B865C829D
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 – 87 fe 53 a7 40 3a 55 e8-fb d1 a2 87 47 63 24 07 ..S.@:U…..Gc$.
0010 – a2 b6 03 40 a3 36 c4 92-d6 55 85 e0 bf eb 8d 00 …@.6…U……
0020 – 3a 27 40 88 c9 92 25 21-0d ae 8a a1 00 bf 68 b0 :’@…%!……h.
0030 – 09 f3 b8 e4 26 57 25 2d-b7 a7 26 ab ad 8e 0a 13 ….&W%-..&…..
0040 – 32 e1 1b 78 f1 df e4 27-49 3c 34 52 0e 02 09 69 2..x…’I<4R…i
0050 – 77 a4 a4 26 9e 26 98 6f-6e 11 4c f1 0b 79 72 a0 w..&.&.on.L..yr.
0060 – 99 26 9e ef 25 53 da 43-33 1e 03 b5 89 ff a7 1a .&..%S.C3…….
0070 – 9f 74 45 26 8e 82 0b 80-dd 5c bc f9 07 c5 d9 c0 .tE&…..\……
0080 – 47 40 db 61 e4 69 0d 85-49 16 8b 55 54 60 69 26 G@.a.i..I..UT`i&
0090 – 11 91 79 65 87 1b e4 8d-f7 ff e6 66 fc 57 51 bc ..ye…….f.WQ.
Start Time: 1644631874
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
Before i open a ticket with MS is there anything stunning I am missing?
I’d open a support call.
We do have similar issue. Do we have any solution?
Hi Kevin,
Is SQL 2019 CU15 officially supported for SCOM 2019 ?
Sushanth S K
We do not comment on specific CU’s unless it is listed in the product documentation as a minimum requirement. For any supported version of SQL, SCOM always supports whatever the currently released CU is.
im getting the “Error: :PopulateUserRoles: failed” on my scom 2019 install and i have already confirmed that the tls 1.2 prereqs are installed per this link: https://docs.microsoft.com/en-us/system-center/scom/plan-security-tls12-config?view=sc-om-2019
i decided to turn on verbose logging on the schannel and i am seeing the below at the time the :PopulateUserRoles errors out:
The certificate received from the remote server was issued by an untrusted certificate authority. Because of this, none of the data contained in the certificate can be validated. The TLS connection request has failed. The attached data contains the server certificate.
A fatal alert was generated and sent to the remote endpoint. This may result in termination of the connection. The TLS protocol defined fatal alert code is 48. The TLS alert registry can be found at http://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-6
the iana link says alert code 48 is “unknown ca” but im not quite sure what certificate is being referenced either on the mgmt server or the db server.
I have not seen TLS 1.2 fail when the prereqs are there. What version did you install? The previous versions are still available for download.
on my MS i have MSOLEDB 19.0 and MSOLEDB 18.3 both installed. and they are also both installed on my DB and DW boxes as well. in addition i also have MS SQL Server 2012 Native Client 11.4 on all 3 servers. the OS is 2019 and the SQL server vrs is 2019
SCOM 2019 did not support MSOLEDB at install.
https://kevinholman.com/2018/05/06/implementing-tls-1-2-enforcement-with-scom/
One caveat – SQL Native Client is deprecated. SNAC11 and ODBC11/13 are the minimum drivers to support TLS 1.2. However, we generally recommend the latest MSOLEDBSQL which is actually supported by SQL.
Our current documentation does not specify that we support MSOLEDBSQL in SCOM 2019, but I believe we started supporting it in SCOM 2019 UR2.
My general recommendation is to go with what works – For SCOM 2019 new deployments – install SNAC11 + ODBC13, then install SCOM 2019, then install the latest Update Rollup, then install MSOLEDBSQL
i removed all versions of MSOLEDB and ODBC from my mgmt server and only left SNAC 11.4 and ODBC 13.0. kicked off another install and now the MS, DB, DW are installed/configured. i appreciate ur time as always man!
I always use a UDL file on the desktop to test and validate TLS being the issue: https://mohammaddarab.com/how-to-test-connection-to-sql-server-using-udl-file/
Hi Kevin,
We updated our company SCOM2019 Mgmt server and Agents, progressively from RTM to UR3.
All SCOM Mgmt servers and the Agents were updated successfully and we can see the UR3 versions inside the SCOM Mgmt server console.
Mgmt server version = 10.19.10505.0
Agents version = 10.19.10177.0 .
However; once you open the monitored server registry, the SCOM Agent is still displaying RTM version = 10.19.10014.0.
We uninstall and re-install the SCOM Agent manually and updated it progressively from RTM to UR3, we can see the SCOM Agent version update itself progressively in the SCOM Mgmt server but does not update its version in the monitored registry it stay on the RTM version = 10.19.10014.0
Why do you think the registry should change? Which key?
Hi Kevin,
Its not the Licence key, its the SCOM Agent Version # in the monitored server registry and the control panel program, that stays on # 10.19.10014.0 even if we update manually or automatically.
it does show the correct version in SCOM Mgmt server
Atif .
(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”)
(Get-ItemProperty “HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\setup”).AgentVersion
I believe once we update the SCOM Agent to UR3 it must change from RTM version = 10.19.10014.0.to Agents version = 10.19.10177.0.
It does show the correct agent version (10.19.10177.0.) in the SCOM2019 Mgmt server conrol panel but inside the monitored server the SCOM Agent remains on RTM version = 10.19.10014.0
Hi Kevin,
Good day.
Is it possible for our SCOM service accounts will be manage GMSA (Group Managed Service Accounts – https://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview)? The AD team is implementing every 100 days, password will automatically change and this will impact our SCOM environment. RunAs accounts are also affected is also affected. Does SCOM 2019 and SCOM 2016 have this capability for the GMSA? Additionally, we still have SCOM 2012 R2 and migrating the agents to 2019. 🙂
Thanks and more power.
Hi Kevin,
I am doing a side by side migration of SCOM 2012R2 to SCOM 2019. Is it possible for me to apply SCOM 2019 UR4 Console/Agent update to my already deployed SCOM 2012 R2 Consoles and Agents so they can be upgraded to SCOM 2019 UR4 version? I have about 40 SCOM 2012 R2 consoles installed on 40 computers and about 500 2012 R2 SCOM Agents deployed to servers all on the same domain.
We use SCCM to deploy Consoles and Agents to servers/workstations, so I was thinking of having SCCM uninstall the SCOM 2012 R2 agents and consoles, then have SCCM reinstall the SCOM agents and consoles with the SCOM 2019 UR4 updates including the Console hotfix. I just didnt know if upgrading was a possibility for the already installed 2012 R2 consoles/agents via SCCM or if I would have to do the uninstall and re-install route?
Thanks in advance, Kevin! Your website has helped me out a ton.
Hi Kevin,
Any help here.. getting below error while discovery linux device from scom 2022.
WinRM cannot process the request.
The following error occurred while using Kerberos authentication: The computer xxx.xxx.xxx is unknown to Kerberos. Verify that the computer exists on the network, that the name provided is spelled correctly, and that the Kerberos configuration for accessing the computer is correct.
The most common Kerberos configuration issue is that an SPN with the format HTTP/xxx.xxx.xxx is not configured for the target. If Kerberos is not required, specify the Negotiate authentication mechanism and resubmit the operation
Hi Kevin,
We’ve completed an in-place upgrade of a 2016 management group to 2019.
The environment consists of two management servers on Windows 2016 server, and
SQL 2016 for our DBs.
As far as I’m aware, we followed the step by step guide to the letter. There were no errors during
the upgrade on either management server.
Following the upgrade, both management servers are in an offline state (grey ticks in console).
The second management server to be upgraded is now displaying an alert the same as the one described
by John in this thread:
“Management configuration service failed to complete bootstrap procedure”. This alert is NOT present
on the first mgmt server.
The body of the same alert reports that the version of cscmdbops.dll is lower than the expected version of
10.19.10050.0.
However when I check the .dll version on each mgmt server it is actually version 10.19.10050.0, not the
version 7.x .dll that the alert is complaining about.
Grateful for any advice. We’re trying to avoid a side by side upgrade, but can roll back to
snap shots for this environment if required.
Thanks,
Christian.
Update: The grey state of the management servers was resolved by giving the SCOM action and Datawarehouse write accounts the local “logon as a service” right on each management server. I missed this requirement in the security matrix spreadsheet.
The bootstrap error persists. Will probably rollback the upgrade and try again with above permissions in place.
Good morning Kevin,
Will using Local System as the action and SDK account cause any issues? My current org would like to use the least amount of accounts for SCOM. We used to have four (One for AA, SDK, DReader, DWriter), but now we want to use LocalSystem for SDK/AA and one account for DReader/DWriter. Is that possible?
In a large environment, you end up with a lot of SQL logins when doing that (for each management server Computer Account) but that’s the biggest hurdle I know of. Why not just use a single account for all of SCOM? I am heading in that direction. I have never understood why we separate into 4 distinct accounts, and nobody has ever provided a good reason for continuing that beyond the “separation of duties with minimal rights” debate. Seems like a lot of extra garbage to keep up with for very little purpose.
Our environment is only 2 Management Servers and we will be managing about 200 or less agents. So it will be a small environment. We are going to convert our SCOM accounts to GMSA today, so I will ask to see if we can just use one account for everything instead of using local system for AA/SDK.
On another note, I think I would have to use at least 1 account for everything. Im going through the steps to switch to GMSA accounts and each account requires different roles within the databases. Im not too savvy with SQL, but If Im using local system as the AA and SDK account, thats gonna be a problem adding roles isnt it? With using one account, I should be able to apply all those roles to that one account and be good to go (I think)
You can use a single GMSA for everything *except* the SSRS report server execution account.
I just had a customer do this. When you add the SQL login, you grant the specific role level requirements as documented here:
https://kevinholman.com/2020/07/23/scom-2019-security-account-matrix/
https://kevinholman.com/2022/09/26/scom-2022-security-account-matrix/
You simply add those specific rights for the single account, based on the 4 individual account/role requirements. No problemo.
It worked like a charm. We are actually using the AA/SDK as local system and the Data Reader/Writer accounts as 1 GMSA. So far no errors in events log or anything abnormal.
Hi Kevin,
Good day.
What will happen if we remove the SCOM service accounts from local admin groups? Our Security is asking us to remove the service accounts from admin group. Appreciate your response