This article covers SQL Agent job monitoring in the current SQL Version Agnostic Management pack, version 220.127.116.11 and later.
I will cover three conceptual areas:
- Out of the box experience and defaults
- SQL Agent only monitoring (monitoring all jobs at the agent level)
- SQL Agent Job monitoring (optionally discovering and monitoring agent jobs on an individual level)
Out of the box, we don’t discover or monitor individual SQL agent jobs by default. What we do is discover the SQL Agent object:
Out of the box, we monitor for Service Running status, Last Job Run Status, and Job Duration:
Last Run Status – checks all jobs on the SQL Agent every 10 minutes, and if ANY job didn’t complete successfully this monitor will stay in a warning state. This does not generate an alert because there is an override to disable alerts to control noise. If you want this level of monitoring, you need to override Alerting back to enabled.
Job Duration – checks all jobs on the SQL Agent every 10 minutes, and if ANY job takes longer than 60 minutes it will set a warning state, or 120 minutes for a critical state. Alerting is also disabled via override. If you want this level of monitoring, you need to override Alerting back to enabled.
There are also several rules that target the “SQL Agent” object, that primarily look in the Application event log for issues/errors related to the SQL Agent:
Notice that the “A SQL job failed to complete successfully” is disabled out of the box. This was done to reduce out of the box noise…. as many customers have terribly monitored/maintained SQL environments and have so many jobs failing that this was almost un-actionable. If you want to be alerted to ANY SQL Agent job failure of any kind – then you might consider enabling this rule via override. I would recommend this if you aren’t going to discover and monitor individual SQL agent jobs (covered later in this article).
Ok – that covers the out of the box defaults, and SQL Agent level monitoring. Now, what if I need to go deeper, and have the ability to override settings for individual SQL Agent jobs? You will notice that in the view for SQL Agent Job State – there are no objects:
This is by design, because we don’t discover SQL Agent jobs individually out of the box. This is because customers have potentially tens of thousands of individual SQL Agent jobs, and discovering these is optional because it can increase load on the monitored agent, and increase instance space and monitoring load in SCOM.
That said, there are scenarios where you might wish to enable this. Perhaps you have SQL Agent jobs that you don’t want to monitor at all, or others that have especially long run-times that need unique thresholds (like SQL backups or maintenance on VERY large DB’s) In this case – we will want to enable the discovery to discover SQL Agent jobs as discovered objects in SCOM.
To enable the object discovery, go to Authoring Pane, select Object Discoveries. In the upper right corner, select Change Scope, then View All, Clear All. Search for “Agent Job”, and select “MSSQL on Windows:Agent Job”
This will show you the discovery we are looking for: “MSSQL on Windows: Discover SQL Server Agent Jobs”
Notice this is disabled by default. Create an override, “For all objects of class: MSSQL on Windows: Agent” and set Enabled = True. Save these overrides to your custom “Overrides – SQL” management pack.
This discovery runs every 4 hours by default, so within 4 hours you should see your SQL Agent job state view populated with your individual SQL Agent jobs. However, you will likely see agent jobs showing up within a few minutes after making the change, if it was never enabled before. (**Hint – to speed this process up in a lab scenario, bounce the Microsoft Monitoring Agent service on your agents – as this will force a discovery to run on service startup)
Now – lets talk about what we monitor by default for discovered SQL Agent jobs. There are a total of two monitors. Last Run Status, and Job Duration:
Last Run Status – monitor checks every 10 minutes, and will remain Yellow (Warning state) for any job whose last run was not a success. This gives you a nice “real time” view of the unhealthy jobs that need some attention.
Job Duration – monitor runs every 10 minutes, and looks for jobs that have exceeded a run-time threshold. The default is 60 minutes (warning) and 120 minutes (critical).
Note*** – On BOTH of the above monitors, alerting is disabled by default to control noise. The out of the box configuration is to show state only. You must override each monitor if you want alerts.
In the following example – I am creating an override for my SQL agent job that performs Index Operations, setting the thresholds higher and enabling alerting….. and saving it to my “Overrides – SQL” management pack:
As you can see, this gives you SQL Agent job by job granularity.
You could easily also create groups of SQL agent jobs based on group name…. if you wanted to treat all jobs with the same or similar names the same from an override perspective. This will additionally ease the burden of maintaining overrides using a dynamic group based on criteria of the job:
See my other SQL MP How-To posts: