One common issue I see regularly, is when customers upgrade their OS in place on Agent monitored systems, or replace and older OS with a new VM and a new OS, but keep the same server name.
This practice causes issues for SCOM. You might see duplicate discovered instances across OS versions when you do this. The “proper” way to manage this, is to delete the machine from SCOM, when performing these OS version upgrades or replacements. However, most of the time, the monitoring team is not even made aware that this activity is happening, and they are left with this issue.
It isn’t terribly harmful, the primary side effect is duplicate alerts when a disk fills up, performance counters collected in duplicate, and increased instance space consumed on the Agent and in SCOM as a whole.
You can detect if you are affected by running this SQL query against your Operations Database:
--BEGIN QUERY SELECT bme.Path, bme.DisplayName, Count(*) AS 'Count' FROM BaseManagedEntity bme WHERE bme.FullName LIKE 'Microsoft.Windows.Server.%.LogicalDisk:%' AND bme.DisplayName = 'C:' GROUP BY bme.Path, bme.DisplayName HAVING Count(*) > 1 --END QUERY
You might have duplication of other OS based versioned objects, but this one will output the names of Computers that have more than one “C:” drive discovered… which usually means it is affected.
You can simply delete these agents from SCOM, then let them come back into the management group as a new agent. Caution however, sometimes when you delete an agent in SCOM and it immediately comes back, it will sit in an “unmonitored” state and require the local Healthservice on the agent get restarted, so plan accordingly.
This will improve over time, as we move more toward “Version Agnostic” management packs, that aren’t so version specific, such as the Windows Server 2016 and Later OS MP, or the SQL 2012+ version agnostic MP.
Unfortunately, the main servers in our environment with this issue are actually SCOM MS from when we migrated them to new VM / OS but kept the names and IPs to retain all of the firewall config and web monitor bindings. Probably not a good idea to delete that agent so we’ve learned to live with it ¯\_(ツ)_/¯
Yeah, that’s a really bad idea to do that. We do not support that. There is no good reason to keep names and IP’s… I understand it makes your life a little easier…. you can ALWAYS flip IP’s later down the road after you build a new MS and want to “cut over”….. that’s no big deal, but the names? The proper way to replace management servers is https://kevinholman.com/2016/02/03/checklist-removing-migrating-old-management-servers-to-new-ones/
So, I was surprised to see a couple of these, but no big deal, but thanks for the post, I will certainly clean this up.
Separately, but still related to disk alerts, I recently realized that a system with a bunch of other drives, D – M are not alerting to low disk space like all the other systems do or that the C drive of that system alerts just fine. The other servers do not have this problem that I can see, where would be the best place to check as to why they are not alerting, but the C is?
SCOM disk space monitoring treats System Drives and Non-System drives differently – with different thresholds. Perhaps this is why you feel you’re not seeing alerts?
Kevin,
Have you ever considered creating an override to disable the old operating system discovery for the affected object and then running Remove-SCOMDisabledClassInstance to clean up the old objects? it would arguably allow you to retain the “new” servers performance and alert history.
Retaining perf and alert history should be retained in the DW, regardless of a quick delete/add back. The managed entity data will be retained, it will simply be marked as deleted, then come back again. I don’t think we actually delete perf/alert data on the spot.
Hi Kevin,
may I ask how “Windows Computer” class is discovered?
I see some reference to Windows Core Library, but I am not able to find out if it is done by some script or how this object is discovered.
Thanks
MF
Windows Computer is a special class that gets created when an agent is approved, much like Healthservice. Then there are normal discoveries to publish class properties. But the initial discovery is inserted.
That makes sense for me.
Thank you for your quick and understandable reply 🙂
Have a nice day
M.
I have in-placed upgraded some SCOM gateway servers, not that I couldnt build new ones but this was much easier and faster.
And yes i have dual disk on them now, in fact the IIS and NICs are duplicates too.
but should it not be possible to clear it out using your own script to remove old clients ?
This one to identify
DECLARE @name varchar(255) = ‘%computer%’
—
SELECT BaseManagedEntityId, FullName, DisplayName, IsDeleted, Path, Name
FROM BaseManagedEntity WHERE FullName like @name OR DisplayName like @name
ORDER BY FullName
and this one to remove the BME IDs ?
DECLARE @EntityId uniqueidentifier = ‘BMEID’
—
DECLARE @TimeGenerated datetime;
SET @TimeGenerated = getutcdate();
BEGIN TRANSACTION
EXEC dbo.p_TypedManagedEntityDelete @EntityId, @TimeGenerated;
COMMIT TRANSACTION
Seems to me it can clean up the extra entries and lave the new ones in the DB
Regards
Kenneth
Yes, that is the only way to clean up machines you cannot easily delete – like GW and MS.
I am working on this same problem only this was not the result of an in place upgrade. It was a new server with a new name.
That’s weird. How did a new server get multiple versions of a disk?
I tried the approach, i could see that the wrong discovery has been removed in console, but in DB still i could see the wrong entry.
Hey Sibi,
i have the same effect. I think this Phenomenon comes from stored data in DB. Maybe in next few days it must be removed automatically? Can you check it a second time?
An alternative method is,
Create a management pack called ‘Custom Delete Duplicate Objects’
In that MP create a group called ‘ CUSTOM Machines with duplicate disks’
Run the following
$Group = Get-SCOMGroup -DisplayName ‘CUSTOM Machines with duplicate disks’
$MPSeal = get-scmanagementpack -name ‘Microsoft.Windows.Server.2012.Discovery’
$Discovery = Get-SCOMDiscovery -managementpack $MPSeal
$MPUnseal = get-scmanagementpack -name ‘ Custom.Delete.Duplicate.Objects’
Disable-SCOMDiscovery -Group $Group -Discovery $Discovery -ManagementPack $MPUnseal
This will disable all the SCOM 2012 discoveries into the new MP.
Now populate your group with Windows Computer objects and wait a few minutes , then run Remove-SCOMDisabledClassInstance
Check health explorer that duplicate disks and Ethernet objects are now gone. This works for agents gateways and MS’s.
Worked perfectly, Thanks a lot.
I also removed later the group members, and the old objects not returned to these servers…
I also had to remove some Windows Server 2008 old objects, which I added to the same group and by repeating the above with this change for disabling 2008 discoveries:
$MPSeal = Get-SCManagementPack -DisplayName ‘Windows Server 2008 Operating System (Discovery)’ # -name ‘Microsoft.Windows.Server.2008.Discovery’
Also, probably Windows 2012R2 servers are also included in the above 2012, as couldn’t find any duplicated R2 after clearing the normal 2012. but here its name in case someone will need it.
$MPSeal = Get-SCManagementPack -DisplayName ‘Windows Server 2012 R2 Operating System (Discovery)’
Hi Kevin
Removing the agents will also remove any custom overrides on them and any membership of groups that those agents been manually added to.
And also manual profile and accounts memberships.
I understand that Andy solution above is a workaround that will disable the old discoveries on those objects..
Even though it kind of messy, does it giid enough and without possible problem? (like disabling needed discoveries for some other products)
Thanks!
No, it will not.
Removing an agent does not impact overrides and groups. Those are object GUID based in management packs and do not change. When you add the agent back, the discovered objects retain the same GUIDs and those will still apply.
Thanks for the answer
I should write might..
I have seen your described behavior most of the times. But also encountered cases which the overrides disappeared after approving the devices back. I’m not sure what made those cases special.
I did Andy solution, which worked perfectly
Hi Kevin, I made a small addition to your query.
We have an automated deployment process for servers that, for test environments, will use the first available name within the namiong convention. That can also be a name that was decomissioned shortly before a new rollout – less than SCOM’s purging routines. As you stated, there can be duplication of other OS based versioned objects – in our case, a LOT of those. Your query, focusing on C:, yielded 56 rows – my updated version below, yielded 410 server names.
What I did first, was create a query looking for all duplicated Logical Disks. That query returned almost 1400 rows, but it contained duplicate server names because a server can have multiple disks. Especially on SQL servers, where we have separate disks for data, log and tempdb. I ahve plenty of servers where the SQL-related disks are duplicated, but C: is not. To get to only server names, we only have to deduplicate 🙂
I put the query into a CTE (Common Table Expression) and then did a SELECT DISTINCT: voila, 410 names.
The query is as follows:
WITH DuplicateObjects AS(
SELECT bme.Path,
bme.DisplayName,
Count(*) AS ‘Count’
FROM BaseManagedEntity bme
WHERE bme.FullName LIKE ‘Microsoft.Windows.Server.%.LogicalDisk:%’
GROUP BY bme.Path, bme.DisplayName
HAVING Count(*) > 1
)
SELECT DISTINCT [Path] FROM DuplicateObjects
As a next step, I’m considering uring your Class.Group.SQLQueryBasedGroupWindowsComputers MP Fragment to create a group for which I can then disable the discovery… time permitting, of course.