One common issue I see regularly, is when customers upgrade their OS in place on Agent monitored systems, or replace and older OS with a new VM and a new OS, but keep the same server name.
This practice causes issues for SCOM. You might see duplicate discovered instances across OS versions when you do this. The “proper” way to manage this, is to delete the machine from SCOM, when performing these OS version upgrades or replacements. However, most of the time, the monitoring team is not even made aware that this activity is happening, and they are left with this issue.
It isn’t terribly harmful, the primary side effect is duplicate alerts when a disk fills up, performance counters collected in duplicate, and increased instance space consumed on the Agent and in SCOM as a whole.
You can detect if you are affected by running this SQL query against your Operations Database:
--BEGIN QUERY SELECT bme.Path, bme.DisplayName, Count(*) AS 'Count' FROM BaseManagedEntity bme WHERE bme.FullName LIKE 'Microsoft.Windows.Server.%.LogicalDisk:%' AND bme.DisplayName = 'C:' GROUP BY bme.Path, bme.DisplayName HAVING Count(*) > 1 --END QUERY
You might have duplication of other OS based versioned objects, but this one will output the names of Computers that have more than one “C:” drive discovered… which usually means it is affected.
You can simply delete these agents from SCOM, then let them come back into the management group as a new agent. Caution however, sometimes when you delete an agent in SCOM and it immediately comes back, it will sit in an “unmonitored” state and require the local Healthservice on the agent get restarted, so plan accordingly.
This will improve over time, as we move more toward “Version Agnostic” management packs, that aren’t so version specific, such as the Windows Server 2016 and Later OS MP, or the SQL 2012+ version agnostic MP.
Unfortunately, the main servers in our environment with this issue are actually SCOM MS from when we migrated them to new VM / OS but kept the names and IPs to retain all of the firewall config and web monitor bindings. Probably not a good idea to delete that agent so we’ve learned to live with it ¯\_(ツ)_/¯
Yeah, that’s a really bad idea to do that. We do not support that. There is no good reason to keep names and IP’s… I understand it makes your life a little easier…. you can ALWAYS flip IP’s later down the road after you build a new MS and want to “cut over”….. that’s no big deal, but the names? The proper way to replace management servers is https://kevinholman.com/2016/02/03/checklist-removing-migrating-old-management-servers-to-new-ones/
So, I was surprised to see a couple of these, but no big deal, but thanks for the post, I will certainly clean this up.
Separately, but still related to disk alerts, I recently realized that a system with a bunch of other drives, D – M are not alerting to low disk space like all the other systems do or that the C drive of that system alerts just fine. The other servers do not have this problem that I can see, where would be the best place to check as to why they are not alerting, but the C is?
SCOM disk space monitoring treats System Drives and Non-System drives differently – with different thresholds. Perhaps this is why you feel you’re not seeing alerts?
Kevin,
Have you ever considered creating an override to disable the old operating system discovery for the affected object and then running Remove-SCOMDisabledClassInstance to clean up the old objects? it would arguably allow you to retain the “new” servers performance and alert history.
Retaining perf and alert history should be retained in the DW, regardless of a quick delete/add back. The managed entity data will be retained, it will simply be marked as deleted, then come back again. I don’t think we actually delete perf/alert data on the spot.
Hi Kevin,
may I ask how “Windows Computer” class is discovered?
I see some reference to Windows Core Library, but I am not able to find out if it is done by some script or how this object is discovered.
Thanks
MF
Windows Computer is a special class that gets created when an agent is approved, much like Healthservice. Then there are normal discoveries to publish class properties. But the initial discovery is inserted.
That makes sense for me.
Thank you for your quick and understandable reply 🙂
Have a nice day
M.
I have in-placed upgraded some SCOM gateway servers, not that I couldnt build new ones but this was much easier and faster.
And yes i have dual disk on them now, in fact the IIS and NICs are duplicates too.
but should it not be possible to clear it out using your own script to remove old clients ?
This one to identify
DECLARE @name varchar(255) = ‘%computer%’
—
SELECT BaseManagedEntityId, FullName, DisplayName, IsDeleted, Path, Name
FROM BaseManagedEntity WHERE FullName like @name OR DisplayName like @name
ORDER BY FullName
and this one to remove the BME IDs ?
DECLARE @EntityId uniqueidentifier = ‘BMEID’
—
DECLARE @TimeGenerated datetime;
SET @TimeGenerated = getutcdate();
BEGIN TRANSACTION
EXEC dbo.p_TypedManagedEntityDelete @EntityId, @TimeGenerated;
COMMIT TRANSACTION
Seems to me it can clean up the extra entries and lave the new ones in the DB
Regards
Kenneth
Yes, that is the only way to clean up machines you cannot easily delete – like GW and MS.
I am working on this same problem only this was not the result of an in place upgrade. It was a new server with a new name.
That’s weird. How did a new server get multiple versions of a disk?
I tried the approach, i could see that the wrong discovery has been removed in console, but in DB still i could see the wrong entry.
Hey Sibi,
i have the same effect. I think this Phenomenon comes from stored data in DB. Maybe in next few days it must be removed automatically? Can you check it a second time?
An alternative method is,
Create a management pack called ‘Custom Delete Duplicate Objects’
In that MP create a group called ‘ CUSTOM Machines with duplicate disks’
Run the following
$Group = Get-SCOMGroup -DisplayName ‘CUSTOM Machines with duplicate disks’
$MPSeal = get-scmanagementpack -name ‘Microsoft.Windows.Server.2012.Discovery’
$Discovery = Get-SCOMDiscovery -managementpack $MPSeal
$MPUnseal = get-scmanagementpack -name ‘ Custom.Delete.Duplicate.Objects’
Disable-SCOMDiscovery -Group $Group -Discovery $Discovery -ManagementPack $MPUnseal
This will disable all the SCOM 2012 discoveries into the new MP.
Now populate your group with Windows Computer objects and wait a few minutes , then run Remove-SCOMDisabledClassInstance
Check health explorer that duplicate disks and Ethernet objects are now gone. This works for agents gateways and MS’s.