Author Archives: Andres

About Andres

I am a Senior Premier Field Engineer at Microsoft. I work with System Center Operations Manager (SCOM) and Azure Monitoring, and used to work with all things middleware. I especially enjoy the creative side of my work, where I help people come up with custom solutions for particular problems.

Moving is such sweet sorrow

Welcome to my new home, or at least the new  home for my System Center Operations Manager (SCOM) and Azure Monitoring.  For reasons too boring to get into, I’ve been forced to re-locate my work to this new space, and hope to leverage this new platform for more frequent posts.
Thank you for following me to my new home, and I look forward to discussing all things Microsoft  monitoring with you.

Advertisement

Useful Reports in System Center Operations Manager

A lot of my customers never use the reporting feature of System Center Operations Manager.  Some do not even install the reporting component, and are happy to simply drift through the river of life without knowing what’s coming around the next bend.  A lot of them have the reporting component of SCOM installed, but either never use it, or think that it is only useful for technology administrators (AD, SQL, IIS, SharePoint, etc.)  Well, I am here to tell you that SCOM reports CAN help the SCOM administrator, and are in fact very useful.
One of my favorite reports is the Data Volume by Management Pack.  Ask yourself this question:  How much do you know about exactly what is going into your Operations Manager databases?  Besides alerts, what else is going inside the databases?  Well, the Data Volume by Management Pack can show you this information, and more.

First, where do we find the Data Volume by Management Pack report?  In the Reporting workspace, it can be found under the category folder called System Center Core Monitoring Reports:

 

image

 

This is actually one of the easiest reports to configure and run in Operations Manager.  When you open the report, you get the following configuration fields:

image

 

First configure the time frame for the report.  Keep in mind that the larger timeframe you configure, the longer the report will take to execute.  I like to run this report on a monthly basis, so configuring a month period in the From and To fields is usually good enough for my needs.  For this example I am going to use 10 days.  The rest of the fields I generally leave unchanged, but you can adjust them as necessary.    One thing I would like to mention, and which I find few people know is in regards to the Management Pack control on the bottom right corner of the screenshot above.  In this control you can check or un-check specific management packs you wish to include or exclude from your report.  By default all management packs are checked, and in fact for this report I usually leave them all checked.  However, if you want to run this report for only a sub-set of management packs, you do not need to manually un-check each management pack.  If you right-click on the control, you will get a context menu which allows you to check or un-check all the items in the control.  This control is also available in other reports, most notably the Most Common Alerts report under the Generic Report Library, but that’s a topic for another day.

image

 

Once you have configured the report settings, and click the run button, you will get output which looks something like this (this is just the first page out of two pages of output):

image

 

You may not realize it now but this report is chuck full of useful information, so let’s take a look.  The first thing I want to point out is the table at the top showing Counts by Management Pack. Notice that the first management pack listed, the Microsoft SQL Server 2012 (Monitoring) management pack accounts for 92.7% of the total volume of data being written into my databases.  I don’t know about you, but to me that sounds like a lot.  In your environment you may see different numbers, but in my experience, the top items in this list are typically all the SQL monitoring management packs across SQL versions.  What does this tell me?  Well, it tells me that if I want to do some tuning of workflows to reduce what goes into my database the SQL 2012 Monitoring management pack is probably a good place to start. 

Notice the columns on the right side of the table.  Discovery Data is low, in my case 0 because few (or in my case none) databases or SQL objects were discovered over the timeframe of the report.  Alert Count does show some alerts, as does the State Changes column.  These indicate that monitors detected problems, probably raised alerts, and hopefully my DBAs did something about them.  What is far more interesting, however, is the Perf Count column.  Over one million performance measurements were captured in a 10 day period in my environment.  If you don’t think that is a lot, let me add this:  I only have one SQL server in my lab.  So from one server, in 10 days, one million SQL related performance measurements were taken.  This does not include Operating System performance measurements like CPU, Disk, Memory, etc. 

As it turns out the SQL management packs have a bunch of performance collection rules, which run on every SQL server, across all versions and collect a variety of SQL specific performance metrics.  This is not necessarily a bad thing, if the SQL Admin team is consuming these performance metrics through reports, dashboards and/or views.  However, in my experience the SQL team either does not consume any of these metrics, or consumes very few of these metrics.  If only there were a way to tell exactly which SQL performance counters are being collected the most.  If only we could tell which rules within the SQL 2012 Monitoring management pack are doing all of these one-million plus performance collections in my environment.  Oh, wait.  There is!

Notice that the contents of each column are the data items but in the form of links.  Links which can be clicked on, and trigger the opening of what are appropriately named Linked Reports.  In this case, when you click on the number (1147086) under the Perf Count column for the Microsoft SQL Server 2012 (Monitoring) row, it triggers the opening of a details report.  The report it opens is actually a pre-configured version of a report called Data Volume by Workflow and Instance (preconfigured with the same settings as your original report: From date, To date, etc.)  The Data Volume by Workflow and Instance report is available under the System Center Core Monitoring Reports as well.  In my example it looks something like this:

image

Notice the Counts by Discovered Type, Rule or Monitor table at the top and how it lists the rules which are collecting the most performance data in my environment.  Most of the workflows are pretty evenly distributed in how much data they collect.  Again, if these data items are needed, then there is not much to do.  However, now that we know what data the SQL management pack’s performance collection rules are inserting into the Operations Manager databases, I can have an honest discussion with my DBAs about what they need and don’t need.  If they tell me they don’t need specific items, I can go override those performance collection rules to disable them. 

I went through this exercise with one of my customers recently, and their DBAs came back with:  “We don’t need any of those performance metrics from SCOM. We don’t use them, we don’t see them, we don’t need them.”  It turns out they use something else to track and chart SQL performance and wanted no SQL performance collections in SCOM.  My customer does want all other SQL monitoring, meaning the rules and monitors which raise alerts.  They just didn’t want the performance data, so they wanted to disable all the SQL performance collection rules across all versions of SQL (2005, 2008, 2012, 2014, 2016).

NOTE:  Yes, it is possible to disable just the performance rules, without disabling any monitors (even monitors which leverage the same performance metrics as rules I disable.)  Performance based monitors will either have their own data source, or a shared data source with the performance rules.  Disabling a rule does not disable the underlying data source so the monitors will be fine either way.

So the question then becomes, how do I disable all the performance based rules in ALL the SQL management packs across all versions and components?  The answer:  PowerShell, of course.  I am a big believer in not reinventing the wheel if I don’t need to.  Sridhar Vishwanatham wrote an excellent article on TechNet titled:  SCOM 2012: Bulk Override Creation for Monitors and Rules using PowerShell.  He has a script in the Rules section which almost did what I needed, so I performed a few quick modifications:

1 import-module operationsmanager 2 3 $mps=Get-SCOMManagementPack |?{($_.name -match "sql") -and ($_.Sealed -eq "True")} 4 $overrideMp= Get-SCOMManagementPack -Displayname "Litware.SQL.Customizations2.MP" 5 $rules=$mps|Get-SCOMRule | ? {$_.Category -eq "PerformanceCollection"} 6 7 8 foreach($rule in $rules) 9 { 10 if($rule.Enabled -ne "false") 11 { 12 $Target= Get-SCOMClass -id $rule.Target.id 13 $overridname=$rule.name+".Override" 14 $override = New-Object Microsoft.EnterpriseManagement.Configuration.ManagementPackRulePropertyOverride($overrideMp,$overridname) 15 $override.Rule = $rule 16 $Override.Property = 'Enabled' 17 $override.Value = 'false' 18 $override.Context = $Target 19 $override.DisplayName = $overridname 20 $Override.Comment = "Disabling as per request from SQL DBAs on " + (Get-Date).Date 21 } 22 } 23 24 $overrideMp.Verify() 25 $overrideMp.AcceptChanges() 26

I ran the script above directly on one of my customer’s Operations Manager Management  servers, and then verified the SQL performance collection rules across all SQL management pack versions were disabled.  I do want to point out that in line number 4 I specify a management pack in which so store the overrides.  I chose to store these overrides on a separate management pack from where I keep all my other SQL customizations, in case I want to reverse these performance collection rule overrides in bulk.  If a time ever comes where the SQL team again wants to collect performance information for SQL using the SQL management packs, all I need to do is remove the management pack called “Litware.SQL.Customizations2.MP” from the SCOM management group, and performance collections will begin anew.

 

After having performed the override of all SQL performance collection rules across SQL versions, running the Data Volume by Management Pack, and configuring a timeframe of a few days AFTER the overrides went into effect, I now get a report like this:

image

 

Notice that far less data is going into my SCOM databases, and I can then choose whether I need to tune any of the workflows from the remaining management packs.  It is important to note that I do not look to get rid of all data going into the SCOM databases.  The objective is to have SCOM monitor the items you care about, and collect data that you care about.  Collecting data which you do not need, and will never use is a waste of time and resources, so the trick is to tune SCOM to collect only what you need, and nothing else. 

 

There is one more useful piece of information you can glean from the Data Volume by Management Pack report.  Below the Counts by Management Pack table, there are some graphs, displaying information about things like:

  • Count of Discovery Data Submitted by Day
  • Count of Alerts Generated by Day
  • Count of Performance Data Samples Collected by Day
  • Count of Events Collected by Day, and
  • Count of State Changes by Day.

 

These graphs can also be useful, albeit in a slightly different way.  When I look at these graphs, I look for things which are out of the ordinary.  Take a look at the following chart from the Data Volume by Management Pack report:

 

image

 

When I look at this graph, my immediate thought is: “what happened on 10/4/2017?  Why are there so many MORE state changes on that date than in all the other dates in the timespan of the report?”  Now, the graph by itself does not indicate a problem.  There are any number of reasons which could explain why this happened.  Maybe I had some new agents deployed that day.  Maybe I imported some new management packs and had not yet tuned them.  Maybe we had some sort of outage we are aware of, but were able to resolve.  Maybe.  But if I cannot find an answer, I may want to dig deeper and find one.  I could re-run the report, but focus just on that date, then see which management pack( s) had monitors changing state a lot that day, and then leverage the Data Volume by Workflow and Instance to narrow things down to specific monitors, and object instances.

 

The Data Volume by Management Pack report and the Data Volume by Workflow Instance report are two handy tools in the toolbox of every great SCOM administrator, and now you too know how to leverage them.

Health Service Modules Event ID 21405

While working with a customer recently we found an odd issue I thought I would share with my readers.  My customer created a new monitor targeted at the Windows Server Operating System class.  The monitor was supposed to run a PowerShell script to collect some registry information and alert if there was anything miss-configured.  The problem they were having is that the script did not seem to be running.
I began to look at the problem, and added code to the script to write an even to the SCOM event log when the script began execution using the MOMAPI COM object.  Sure enough, nothing got logged.  Things got stranger, when I looked at the health explorer for the Computer object in question, as the Operating System rollup monitor and all its underlying aggregate and unit monitors were marked as empty circles.  Essentially, the Operating System, the hardware, and almost everything on the server was not being monitored.
I first suspected it might be some sort of override gone wild.  We searched through the list of overrides in the Authoring workspace of the SCOM console, but came up empty:  no interesting suspects here.
I then had my customer flush the agent cache manually, by stopping the Microsoft Monitoring Agent Service, deleting the contents of the Agents Health Service State folder, and re-starting the service.  I felt the normal wave of relief when Event ID 1201 showed up in the Operations Manager event log, indicating we were successfully communicating with the Management Group.  But my relief soon turned to dread when a sea of warning events began to flood the event log.  The events looked like this:

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          9/6/2017 2:57:05 PM
Event ID:      21405
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      SERVERNAME.DOMAINNAME.COM
Description:
The process started at 2:57:05 PM failed to create System.Discovery.Data, no errors detected in the output.  The process exited with 1
Command executed:     “C:Windowssystem32cscript.exe” /nologo “PopulateAdvisorWindowsServerProperties.vbs” {DE47446F-B2E9-AFFC-D672-CD274318C360} {116028CA-EEA1-074A-3425-70EEC8705455}
Working Directory:          C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateMonitoring Host Temporary Files 42185
One or more workflows were affected by this. 

The name of the script changed from event occurrence to event occurrence, but it looked like pretty much every .VBS script and .js script was failing to execute on the server.  As a result, the Windows Operating System instance for the server was not being successfully discovered, nor any of the object instances derived or dependent on that class.  My customer’s new custom monitor was simply a victim of the problem here.

One of my strategies when SCOM scripts fail to execute, is to try to run them myself, manually from a command prompt.  Why not?  We have the path to the script, and even the parameters it takes, right from the event log description:
Path:  C:Program FilesMicrosoft Monitoring AgentAgentHealth Service StateMonitoring Host Temporary Files 42185

Syntax:  cscript.exe /nologo PopulateAdvisorWindowsServerProperties.vbs {DE47446F-B2E9-AFFC-D672-CD274318C360} {116028CA-EEA1-074A-3425-70EEC8705455}

(Keep in mind the GUIDs in the command or switches in your instance may be different.)

When we tried to run the script we got an error:
Input Error: There is no script engine for file extension “.vbs”.

A similar error occurred when trying to execute a .JS script.
Fortunately the solution is simple, and quick.  All you have to do is re-register the script file types so the Operating System knows how to execute them.  This can be done with the commands:

assoc .vbs=VBSFile

and

assoc .js=JSFile

After we ran these two commands, we gave the SCOM Agent another manual flush, and when we re-started the Microsoft Monitoring Agent service, we were no longer inundated with event ID 21405.  The scripts ran exactly as they were supposed to, the Operating System and its underlying objects were successfully discovered, and my customer’s custom PowerShell script monitor ran exactly as designed.

SCOM Connected Management Groups–2016 and 2012

System Center Operations Manager contains a little used feature called Connected Management Groups.  Essentially, it allows administrators to see alerts across multiple SCOM Management Groups from one console, side by side.  When you have more than one Management Group, and are responsible for seeing or managing open alerts across them, this can be pretty handy.  The feature has been around for a while, and though I don’t see many of my customers use it (they should!), it is a viable solution in many scenarios.

Recently I had a customer stand up a new SCOM 2016 environment for a proof of concept, and they wanted to know if they could connect their new management group with their existing SCOM Lab environment.  The wrinkle was that the existing Lab environment runs SCOM 2012 R2.  Although the documentation in TechNet states that connected management groups are only supported when the two groups are running the same version of SCOM, we decided to throw caution to the wind and try it out.  It turns out SCOM 2012 R2 and SCOM 2016 do play nice with each other.

So how do you connect them?

First of all, you need to log on to the SCOM Administration Console with SCOM Administrator privileges, and navigate to the Administration Workspace.  The first item on the Administration Pane is easy to miss, but it is what we are looking for:  Connected Management Groups.

image

Click the Add Management Group item on the task pane on the right to configure the management group to connect to.

In the Add Management Group dialog box, enter the name of the other SCOM Management Group you want to connect to, and the Fully Qualified Domain Name of a management server in that Management Group (any of the management servers will do, so long as the System Center Data Access service is up and running).

In the security section, you can leverage the existing SDK account (the account under which the System Center Data Access service runs), or you can specify different credentials using the “Other user account” radio button.

image

After clicking OK, you will see the configured Management Group in the middle pane of the SCOM Administration console:

 

image

 

To view the alerts from the newly connected Management group, you will have to navigate to the Monitoring workspace, and click the Active Alerts view.  There is now a new button available in the button ribbon at the top of the Administration console labeled Show Connected Alerts.  In the screenshot below, notice how there are initially no alerts in my new SCOM 2016 management group:

image

 

After clicking on the Show Connected Alerts button, we get a dialog box prompting for credentials for the target Management Group.  After supplying credentials with permissions to the target group, the Active Alerts view now looks like this:

 

image

 

As you can see, the SCOM 2016 environment is now showing me alerts from both environments (2012 and 2016), side by side, in one pane of glass.  This should help reduce the complexity of managing alerts for users responsible for responding to alerts across multiple SCOM Management Groups.  Although the support documentation states both management groups should be at the same version level, I have confirmed the Connected Management Group functionality works even across versions from 2012 to 2016.

Of course, the one draw-back is that a Connected Management Group allows you to see the alerts from multiple Management Groups in one pane of glass, but it does not let you manage them side by side.  For that you still have to connect to each Management Group separately, one at a time.

Searching through SCOM 2012 Notification Subscriptions using PowerShell.

Imagine a scenario where you have more than 270 notification subscriptions configured in System Center 2012 Operations Manager.  Then, out of the blue you get a request from someone else at your company wanting to know why they are not getting e-mails about the servers they manage, but other people are.  You need to find out what notification subscriptions exist for these servers, so that you can add the appropriate administrator as a recipient.

The problem is that searching through the configuration of notification subscriptions is typically a manual task which can be difficult, and time time consuming.  So the question is how to automate, and speed up the process of searching through subscriptions to find any references to the servers in question.  The answer, of course, is PowerShell.

I created a script which performs the following actions:

1. Inputs

The script takes a list of one or more server names into the $computers variable using fully qualified domain names.

2. Searches

First the script searches and compiles a list of all groups the computers are members of, and searches for any subscriptions which contain references to the groups found.

Then it searches for notifications subscriptions referring to the servers by using the servers name specifically.

Finally, it queries the notification subscriptions for all objects related to the server name.  This ancillary object search accounts for things like Logical Disks, Network Adapters, the Windows Operating System, IIS Web Server role, IIS web sites, IIS Application Pools, DNS Server roles, SCOM Health Service, etc.

3. Output

The script then outputs a list of any and all found notification subscriptions which contain references to the searched servers, objects they contain, or groups which contain the servers.

 

Caveats

There had to be a downside, right?  Well, in this case the downside is performance.  The script execution may take a long time, especially if the management group has a large number of groups and/or a large number of notification subscriptions.  Additionally, this script does not account for the possibility of notification subscriptions based on a specific class, as such a search would be far too complex to execute.

Without further ado:

 

1 #Script searches for SCOM Notification Subscriptions to find which subscriptions a computer might be referenced in 2 #Subscriptions by classes are not covered here. 3 #Written by: Andres Naranjo 4 #Provided as a sample only. 5 6 $subscriptions = get-scomnotificationsubscription 7 $computers = @("sqllit.litware.com","rmslit.litware.com","sqllit.litware.com") 8 $groupMasterList = @() 9 $global:subscriptionsMasterList = @() 10 11 Function searchSubscriptionCriteria 12 { 13 Param ($searchString) 14 foreach ($subscription in $subscriptions) 15 { 16 if ($subscription.configuration.criteria.length -ne 0 ) 17 { 18 if ($subscription.configuration.criteria.Contains($searchString)) 19 { 20 #write-host $subscription.DisplayName -foreground "red" 21 $global:subscriptionsMasterList += $subscription.DisplayName 22 } 23 } 24 } 25 } 26 27 Function searchSubscriptionGroups 28 { 29 Param ($searchString) 30 foreach ($subscription in $subscriptions) 31 { 32 if ($subscription.configuration.MonitoringObjectgroupIds -ne $null) 33 { 34 foreach ($item in $subscription.configuration.MonitoringObjectGroupIds) 35 { 36 if ($item.Guid -eq $searchString) 37 { 38 #write-host $subscription.DisplayName -foreground "red" 39 $global:subscriptionsMasterList += $subscription.DisplayName 40 } 41 } 42 } 43 } 44 } 45 46 Function searchForGroupMembership 47 { 48 Param($searchString) 49 $groups = get-scomgroup 50 51 foreach ($group in $groups) 52 { 53 $members = $group | get-scomclassinstance 54 foreach ($member in $members) 55 { 56 if (($member.Path -eq $searchString) -or ($member.DisplayName -eq $searchString)) 57 { 58 $groupMasterList += $group.DisplayName 59 break; 60 } 61 } 62 } 63 64 } 65 66 foreach ($computer in $computers) 67 { 68 "For Computer $computer : " 69 #Search for groups the computer is a member of 70 "Searching group memberships." 71 searchForGroupMembership($computer) 72 #searching for the groups in all notifications 73 "Searching groups in notifications." 74 foreach($grp in $groupMasterList) 75 { 76 $Id = (get-scomgroup -DisplayName "$grp").Id 77 searchSubscriptionGroups($Id) 78 } 79 80 #searching for subscriptions by FQDN computer name 81 "Searching notifications for computer objects." 82 $computerGUIDs = get-scomclassinstance -displayname "*$computer*" 83 foreach ($cGUID in $computerGUIDs) 84 { 85 searchSubscriptionCriteria($cGUID.Id) 86 } 87 88 #searching for contained classes of the object 89 "Searching notifications for contained/ancillary objects" 90 $containedGUIDs = get-scomclassinstance | where-object {$_.Path -eq "$computer"} 91 foreach($containedGUID in $containedGUIDs) 92 { 93 searchSubscriptionCriteria($containedGUID.Id) 94 } 95 } 96 97 write-host "The following subscriptions reference the specified computers or their objects:" -ForegroundColor "red" 98 $global:subscriptionsMasterList | select -Unique 99

Which Notifications does “Lola” subscribe to?

Recently I ran into an issue where I needed to enumerate the SCOM subscriptions a user subscribes to.  The management group in question has close to 250 subscriptions, and over 60 subscribers.  When someone (let’s call her “Lola”) leaves the company, it is now cumbersome to check which subscriptions should be re-configured for a new subscriber. A quick search on-line did not yield any results so I thought I would tackle it. 

Clearly this should be possible to do with PowerShell, and sure enough there are System Center 2012 Operations Manager commands which deal with notifications.  Running the command:

image

Yields a list of all the Operations Manager commands, including these 3 dealing with notifications:

image

The command we are looking for is Get-SCOMNotificationSubscription.  We can get help for the command with the aid of the Get-Help command, but there seems to be no built-in method to extract just the subscribers.

Making a call to the function is useful, in that it lets us see the structure and content of the data returned:

image

 

However, if I want to see all the subscriptions which have “Lola” as a recipient, the command by itself is not very useful.  I tried using some “where-object” filters, without success.  Not only does it not work, it does not fail with an error message either:

image

The “ToRecipients” field looked like a string containing the list of recipients, but clearly that can’t be right if I can’t use string comparison operators such as “-Like”, “-Contains” and “-Matches”.  Then I remembered that I can see the data type for the “ToRecipients” by using the “Get-Member” command in PowerShell:

image

As it turns out “Get-Member” is our friend.  The output above shows me that the “ToRecipients” property field is an IList Collection, which can be iterated through using one of the most useful PowerShell constructs: “foreach.”  So I put together a simple script:

1 import-module operationsmanager 2 $nameToFind = "Lola" 3 $subscriptions = get-scomnotificationsubscription 4 foreach ($s in $subscriptions){ 5 foreach ($r in $s.ToRecipients){ 6 if ($r.Name -eq $nameToFind){ 7 $s.DisplayName 8 } 9 } 10 }

 

When I run the script, I get  a list of all subscriptions that “Lola” subscribes to:

image

 

One final item is, how do I know what to put in the script variable above called “$nameToFind”?  Easy.  In the Administration workspace of the Operations Manager administration console, navigate to Subscribers.  Find the subscriber you are interested in, and open up its properties.  The name to use in the script is the text in the Subscriber Name field:

image

 

Now we can easily obtain a list of subscriptions we need to modify when someone leaves the company or changes roles within the organization.

Monitoring for GPO Modifications in SCOM: A Custom Composite Module Approach

Recently one of my customers came to me for help implementing monitoring of GPO Modifications using System Center 2012 R2 Operations Manager (SCOM). My customer pointed me to a great blog article written by Nathan Olmstead entitled: Monitoring for GPO Modifications with SCOMNathan Olmstead’s article updated a previously existing article for SCOM 2007 from Jan Varšavský entitled SCOM: Monitoring GPO Changes using SCOM and PowerShell. Both articles follow essentially the following logic:

1. Create a SCOM rule (Rule #1) to listen for Event ID 5136, 5137 or 5141 in the Security Event Log of domain controllers, and raise an alert.

2. Create a command-line channel which listens for alerts from Rule #1, and kicks of a PowerShell script to dig up more information on the GPO changed.

3. The PowerShell script closes the existing alert from Rule #1, and writes a new event log entry (custom Event ID 11101) in the Operations Manager Event Log.

4. Create a new SCOM rule (Rule #2) to listen for the new event 11101 and generate a new SCOM alert with the new GPO change information.

The approach is clever, and I would like to praise both writers for a great take on how to implement this very useful bit of monitoring. However, I feel that the use of a command-line channel to tie together two separate SCOM rules seemed less than efficient. Why not do everything with one workflow, and generate the needed alert directly? Of course, to listen for an event log entry, then execute a PowerShell script using data from the event, and then raise an alert accordingly requires the use of a custom composite module. This means management pack development, and getting our hands dirty with XML.

Fortunately there are some great resources available which provided me guidance in solving this problem. The Operations Manager Management Pack Authoring Guide created by Brian Wren is a great starting point, especially is article: Creating a Monitor Based on a Custom Module. In addition, Kevin Holman (THE Kevin Holman) recently published an equally useful article: How to monitor for event logs and use a script to modify the output – a composite data source. Kevin’s article uses VB Script, and the SCOM 2007 Authoring Console, both of which I wanted to avoid using, opting instead for PowerShell and Visual Studio 2015 with the Visual Studio Authoring Extensions (VSAE) for SCOM.

I like Brian Wren’s approach of giving step by step instructions and as I myself am learning more about developing management packs with the VSAE I thought I would put together step by step instructions for developing a management pack which creates a custom composite module to listen for GPO changes in the event log, running a PowerShell script with data from the event log, and then raising an alert in SCOM.

DISCLAIMER:  The below example uses a PowerShell script which calls the Get-GPO command.  Get-GPO is only available on Windows Server 2008 R2 and above, and is not available on domain controllers running Windows Server 2008 or Windows Server 2003.

Without further ado:

Create a project and add a reference

1. Open Visual Studio 2015 on a workstation where you have install the System Center Operations Manager Visual Studio Authoring Extensions.

2. Click on the File menu item and select New-> Project.

3. In the New Project dialog box, select Management Pack from the Templates pane on the left.

4. In the middle pane select “Operations Manager 2012 R2 Management Pack” as the project type.

image

5. Enter the project name in the Name text box. I named mine GPOMonitoring, then click OK.

The new Solution will consist of a single project also named GPOMonitoring. We need to add a reference to the Active Directory Library Management pack as our workflows will be targeted at Domain Controllers:

6. In the Solution Explorer window, right-click on the References folder item and select Add Reference

7. In the Add Reference dialog box, select the Browse tab, and navigate to a directory where you have downloaded and extracted the Active Directory Management Pack for SCOM. Select the management pack called Microsoft.Windows.Server.AD.Library.mp:

image

8. Click OK.

Your project will now have the following references:

image

 

Add the PowerShell script

With the project created and the references sorted out, it is now time to start creating the artifacts we need, starting with the PowerShell script:

1. In the Solution Explorer window, right-click the project name (NOT the solution) and select Add->New Item.

2. In the Add New Item dialog box, make sure the Management Pack node is expanded on the left pane and select Resources.

3. In the middle pane select PowerShell script file.

4. In the Name text control at the bottom type the name of your PowerShell script. I named mine CorrelateGPO.ps1.

image

5. Click Add to add a blank PowerShell script file to our project.

6. Add the following PowerShell code to the new empty PowerShell script file:

 

1 param($gpoGUIDRaw,$gpoDomainRaw,$gpoUserRaw) 2 3 function debug($msg) 4 { 5 #Function to write debug information to disk. Modify path below per your needs. 6 [string]::Format("{0} {1}",[datetime]::Now,$msg) | Out-File -FilePath C:tempCorrelateGPO.txt -Append 7 } 8 $error.Clear() 9 #debug("Starting Execution of CorrelateGPO.ps1 " + $error[0]) 10 #debug("Initial Variables - gpoGUIDRaw: " + $gpoGUIDRaw + ", gpoDomainRaw: " + $gpoDomainRaw + ", gpoUserRaw: " + $gpoUserRaw + " . Error: " + $error[0]) 11 $api = new-object -comObject 'MOM.ScriptAPI' 12 $api.LogScriptEvent("CorrelateGPO.ps1", 1701, 2, "PowerShell Script to correlate GPO changes ran.") 13 $bag = $api.CreatePropertyBag() 14 15 #The following lines parse the input values to extract the data we need: 16 $gpoGUID = $gpoGUIDRaw.Split('{}')[1] 17 $gpoDomain = $gpoDomainRaw.Split('"')[1] 18 $gpoUser = $gpoUserRaw.Split('"')[1] 19 20 $gpoName = Get-GPO -Guid $gpoGUID -Domain $gpoDomain 21 22 $Text = "A Group Policy Has been Modified`n User: $($gpoUser)`n GPO Domain: $($gpoDomain) `n GPO Name: $($gpoName.DisplayName) `n GPO GUID: $($gpoGUID) `n Event Description: $($AlertDescription)" 23 24 $bag.AddValue('gpoGUID', $gpoGUID) 25 $bag.AddValue('gpoDomain', $gpoDomain) 26 $bag.AddValue('gpoUser', $gpoUser) 27 $bag.AddValue('text', $Text) 28 $bag.AddValue('status', "OK") 29 30 $bag 31 32 33

 

The script takes 3 parameters: the GUID of the GPO which was changed, the name of the domain and the name of the user who made the GPO change.

The debug function is useful when trying to debug the script to figure out what it is doing or whether it is doing it correctly. Once the script is working, invocations of the debug method can be commented out or removed.

The script parses out the three input variables to remove unnecessary characters or string information. It then uses the Get-FPO PowerShell command to obtain information about the GPO which was modified.

Finally, we build the output in the form of a property bag and return it to the calling workflow. A property bag is simple a data structure, a collection of Name and Value pairs, which we can use in discoveries, rules, and monitors to return data to Operations Manager.

Learning Notes – Part 1

While I am not what you would call a management pack development expert, nor a PowerShell expert, I have learned a few lessons I would like to share with the community about both. The Learning Notes section will highlight any additional lessons I have learned the hard way and which I would like others to learn from.

I. the last line of the script returns the property bag which was instantiated and populated during the rest of the script. Make sure you add a few new lines AFTER the property bag return statement in the final line ($bag). Without the extra newlines the script continues to execute and doesn’t realize it needs to return execution flow to the calling workflow. This applies to scripts in discoveries, rules, monitors, you name it. Notice in the screenshot of my Visual Studio window below how I have lines 31-34 empty. They are there, just a carriage return:

image

 

II. ALWAYS test your script by itself, from a PowerShell command line. It is far easier to debug the script when running it outside of the control of SCOM than when it is executed by SCOM.

With the script ready to go, we need to execute the script, which means we need to define the workflow that will execute it.

 

Add a Probe Action Module Type

We will first add a Management Pack Fragment to our project which will let us write the XML necessary for our custom workflows:

1. In the Solution Explorer, right-click on the Project GPOMonitoring (NOT the solution) and select Add->New Item.

2. In the Add New Item dialog box, in the left pane select Code.

3. In the middle pane select Empty Management Pack Fragment.

4. Enter the name of the fragment in the Name text box at the bottom. I named mine GPOCorrelationModules.mpx:

image

 

This will add a new code document to our GPOMonitoring management pack, which will be empty except for the opening and closing <ManagementPackFragment> XML tags. All the management pack XML we will write will go inside these tags, and will determine the behavior and look of our workflows.

We will begin by adding a Probe Action Module Type to our management pack, which executes the PowerShell Script. In the space between the opening and closing <ManagementPackFragment> tags, paste the following code:

 

1 <TypeDefinitions> 2 <ModuleTypes> 3 4 <ProbeActionModuleType 5 ID="GPOMonitoring.ProbeActionModule.GPOCorrelationScript" 6 Accessibility="Public" 7 Batching="false" 8 PassThrough="false"> 9 <!--The module, when called, will receive 3 string parameters:--> 10 <Configuration> 11 <xsd:element minOccurs="1" name="gpoGUID" type="xsd:string" /> 12 <xsd:element minOccurs="1" name="gpoDomain" type="xsd:string" /> 13 <xsd:element minOccurs="1" name="gpoUser" type="xsd:string" /> 14 </Configuration> 15 <ModuleImplementation Isolation="Any"> 16 <Composite> 17 <MemberModules> 18 <!--The module will execute a PowerShell Script which returns a Property Bag data structure--> 19 <ProbeAction ID="Script" 20 TypeID="Windows!Microsoft.Windows.PowerShellPropertyBagProbe"> 21 <ScriptName>CorrelateGPO.ps1</ScriptName> 22 <ScriptBody>$IncludeFileContent/CorrelateGPO.ps1$</ScriptBody> 23 <!--This block passes the necessary values for GPO GUID, Domain and User to the script as parameters--> 24 <Parameters> 25 <Parameter> 26 <Name>gpoGUID</Name> 27 <Value>$Config/gpoGUID$</Value> 28 </Parameter> 29 <Parameter> 30 <Name>gpoDomain</Name> 31 <Value>$Config/gpoDomain$</Value> 32 </Parameter> 33 <Parameter> 34 <Name>gpoUser</Name> 35 <Value>$Config/gpoUser$</Value> 36 </Parameter> 37 </Parameters> 38 <TimeoutSeconds>300</TimeoutSeconds> 39 </ProbeAction> 40 </MemberModules> 41 <!--The below block dictates this custom type only executes one module with identifier "Script" aka the PowerShellPropertyBagProbe above--> 42 <Composition> 43 <Node ID="Script"/> 44 </Composition> 45 </Composite> 46 </ModuleImplementation> 47 <!--The module outputs a property bag data type, and receives Base data as input--> 48 <OutputType>System!System.PropertyBagData</OutputType> 49 <InputType>System!System.BaseData</InputType> 50 </ProbeActionModuleType> 51 </ModuleTypes> 52 </TypeDefinitions>

 

I have added comments in the most important sections of the module to explain its behavior.

Learning Notes – Part 2

III. In the above XML the parameters passed to the script are passed using the syntax: $Config/gpoGUID$ or $Config/gpoDomain$ or $Config/gpoUser$. The “Config” portion of the syntax is XPATH syntax referring to the Configuration XML block near the top of the module definition. The Config, Data, Target and MPElement variables are well documented in the TechNet article appropriately titled: Variables.

IV. Learn from my mistake. After I wrote this management pack, it would not work as expected. For some reason, when I attempted to call my script, the script would run but it would not receive my input values from the $Config/gpoGUID$ or $Config/gpoDomain$ or $Config/gpoUser$ elements. If I manually passed the data without using the $Config variable, it worked, but not with the $Config element. Major Kudos go to Kevin Holman who pointed out that I was using the variable with a lower case “c” as in $config. Apparently a capital letter makes all the difference in the world. So, when using these variables make sure you use $Config, $Data, $Target, etc; with a capital letter in front.

 

Add a New Data Source Module Type

With the Probe Action Module Type which runs our PowerShell script defined, we now need to integrate it with the event log workflow which listens for the events indicating a GPO modification has been made. For this purpose we must build a new Data Source composed of an event log module, and the PowerShell module we just defined.

1. In between the opening <ModuleTypes> tag, and the <ProbeActionModuleType> tag, add the following XML definition.  Once again I have commented the XML itself to explain its behavior:

1 <DataSourceModuleType ID="GPOMonitoring.Event.DS" 2 Accessibility="Public" 3 Batching="false"> 4 <Configuration> </Configuration> 5 <ModuleImplementation Isolation="Any"> 6 <!--This is a composite workflow consisting of two modules, first an EventProvider module, which triggers our ProbeAction PS module--> 7 <Composite> 8 <MemberModules> 9 <!--The event log module will listen in the Security Event log for for Event ID 5136, 5137, and 5141 with groupPolicyContainer in the event description--> 10 <DataSource ID="Event" TypeID="Windows!Microsoft.Windows.EventProvider"> 11 <ComputerName/> 12 <LogName>Security</LogName> 13 <Expression> 14 <And> 15 <Expression> 16 <SimpleExpression> 17 <ValueExpression> 18 <XPathQuery Type="String">Params/Param[11]</XPathQuery> 19 </ValueExpression> 20 <Operator>Equal</Operator> 21 <ValueExpression> 22 <Value Type="String">groupPolicyContainer</Value> 23 </ValueExpression> 24 </SimpleExpression> 25 </Expression> 26 <Expression> 27 <Or> 28 <Expression> 29 <SimpleExpression> 30 <ValueExpression> 31 <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery> 32 </ValueExpression> 33 <Operator>Equal</Operator> 34 <ValueExpression> 35 <Value Type="UnsignedInteger">5136</Value> 36 </ValueExpression> 37 </SimpleExpression> 38 </Expression> 39 <Expression> 40 <SimpleExpression> 41 <ValueExpression> 42 <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery> 43 </ValueExpression> 44 <Operator>Equal</Operator> 45 <ValueExpression> 46 <Value Type="UnsignedInteger">5137</Value> 47 </ValueExpression> 48 </SimpleExpression> 49 </Expression> 50 <Expression> 51 <SimpleExpression> 52 <ValueExpression> 53 <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery> 54 </ValueExpression> 55 <Operator>NotEqual</Operator> 56 <ValueExpression> 57 <Value Type="UnsignedInteger">5141</Value> 58 </ValueExpression> 59 </SimpleExpression> 60 </Expression> 61 </Or> 62 </Expression> 63 </And> 64 </Expression> 65 </DataSource> 66 <!--The Data Source will call the second module, our custom PowerShell probe--> 67 <ProbeAction ID="Probe" TypeID="GPOMonitoring.ProbeActionModule.GPOCorrelationScript"> 68 <!--SCRIPTPARAMETERS - pass parameters to the script module--> 69 <gpoGUID>"$Data/Params/Param[10]$"</gpoGUID> 70 <gpoDomain>"$Data/Params/Param[7]$"</gpoDomain> 71 <gpoUser>"$Data/Params/Param[4]$"</gpoUser> 72 </ProbeAction> 73 </MemberModules> 74 <!--The composition block dictates the order of module execution in the Data Source.--> 75 <!--First we listen for the event log event, and if found we trigger the PowerShell probe--> 76 <Composition> 77 <Node ID="Probe"> 78 <Node ID="Event"/> 79 </Node> 80 </Composition> 81 </Composite> 82 </ModuleImplementation> 83 <!--We will return the output of the PowerShell Probe as a Property Bag data structure of Name-Value pairs--> 84 <OutputType>System!System.PropertyBagData</OutputType> 85 </DataSourceModuleType>

 

Learning Notes – Part 3

V. Note that the Probe Action module above is of the custom Probe Action Module type we defined earlier.  Further, note that the Probe Action Module passes 3 parameters to the PowerShell script.  I could not find any samples out there which described how to pass multiple parameters like this, so I wondered whether it was possible.  Brian Wren’s samples only pass one parameter, the Machine Name.  When I ran into difficulties passing data to my script I questioned whether multiple inputs were possible.  Well, they are!

VI. In the Probe Action we pass 3 pieces of information to the script.  The 3 pieces of information (gpoGUID, gpoDomain and gpoUser) are obtained from the data string of the event which was just processed by the Event module, and which triggered the execution of the Probe Action module.  Note the syntax of the values: $Data/Params/Param[10]$$Data refers to information in the workflow’s data stream, as described in the previously mentioned TechNet article entitled Variables.  The construct is useful enough to warrant mentioning again. Params/Param[10] refers to the tenth parameter in the event description, in this case containing information about the GUID of the GPO modified.

With the Data Source workflow defined, we can now call it from a SCOM Rule or from a Monitor.  I choose to use a Rule here.  Why?  Well, when a GPO is modified my customer wanted to know about it, but did not need to affect the health of any object.  An Alert Generating Rule is good enough for our purposes.

 

Add an Alert Generating Rule

1. After the closing </TypeDefinitions> tag, and before the closing </ManagementPackFragment> tag, insert the following XML code for an alert generating rule:

 

1 <Monitoring> 2 <Rules> 3 <!--Define a new rule, which will use our custom Data Source. The new rule is targetted at Domain Controllers--> 4 <Rule ID="GPOMonitoring.EventAndScript.Rule" 5 Enabled="true" 6 Target="MWSAL!Microsoft.Windows.Server.AD.DomainControllerRole" 7 ConfirmDelivery="true" 8 Remotable="true" 9 Priority="Normal" 10 DiscardLevel="100"> 11 <Category>Custom</Category> 12 <DataSources> 13 <DataSource ID="DS" TypeID="GPOMonitoring.Event.DS"> 14 </DataSource> 15 </DataSources> 16 <WriteActions> 17 <!--When the desired event is found and the script executes, raise an alert, with the following parameters--> 18 <WriteAction ID="Alert" TypeID="Health!System.Health.GenerateAlert"> 19 <Priority>1</Priority> 20 <Severity>2</Severity> 21 <AlertMessageId>$MPElement[Name="GPOMonitoring.EventAndScript.Rule.GPOChange.AlertMessage"]$</AlertMessageId> 22 <AlertParameters> 23 <AlertParameter1>$Data/Property[@Name='gpoGUID']$</AlertParameter1> 24 <AlertParameter2>$Data/Property[@Name='gpoDomain']$</AlertParameter2> 25 <AlertParameter3>$Data/Property[@Name='gpoUser']$</AlertParameter3> 26 <AlertParameter4>$Data/Property[@Name='text']$</AlertParameter4> 27 <AlertParameter5>$Data/Property[@Name='status']$</AlertParameter5> 28 </AlertParameters> 29 </WriteAction> 30 </WriteActions> 31 </Rule> 32 </Rules> 33 </Monitoring>

 

Once again I have commented the XML to explain its behavior.  The one aspect worth expanding on is the use of the <AlertMessageID> tag, which refers to a segment of XML code we have not yet added: the Language Packs section.

Define the Presentation and Language Packs sections

1. Add the following XML block AFTER the closing </Monitoring> tag which defined the rule in the last section, and before the closing </ManagementPackFragment> tag:

 

1 <Presentation> 2 <StringResources> 3 <StringResource ID="GPOMonitoring.EventAndScript.Rule.GPOChange.AlertMessage"/> 4 </StringResources> 5 </Presentation> 6 <LanguagePacks> 7 <LanguagePack ID="ENU" IsDefault="true"> 8 <DisplayStrings> 9 10 <DisplayString ElementID="GPOMonitoring.EventAndScript.Rule.GPOChange.AlertMessage"> 11 <Name>GPO Change Occurred</Name> 12 <Description>GPO Change Information- 13 GPO GUID: {0} 14 GPO Domain: {1} 15 User: {2} 16 Details: {3} 17 18 </Description> 19 </DisplayString> 20 <DisplayString ElementID="GPOMonitoring.Event.DS"> 21 <Name>GPO Change Event then run correlation script DS</Name> 22 <Description/> 23 </DisplayString> 24 <DisplayString ElementID="GPOMonitoring.EventAndScript.Rule"> 25 <Name>GPO Change Event then run correlation script Rule</Name> 26 <Description/> 27 </DisplayString> 28 </DisplayStrings> 29 </LanguagePack> 30 </LanguagePacks>

 

This block of XML provides user friendly, readable strings for the alert rule’s alert details section, as well as a name for the rule itself.  These strings are displayed in the Operations Manager Administration console when the alert is raised.

 

Build and Deploy

The management pack is ready for use:

1. From the Build menu at the top select Build Solution, or click Ctrl+Shift+B.

2. Resolve any build errors and rebuild the solution until you get output like this:

image

 

You may encounter some Warnings or Messages in the Error List window of Visual Studio, but as long as they are not errors you should be fine.  In my case I get the 67 messages as in the following screenshot, but my management pack works flawlessly:

image

 

The management pack itself will be generated as a .mpb file or a management pack bundle file.  You can find the file in your project’s directory in the “bin” folder, as either a debug or release build:

image

You can import the management pack bundle file directly in to SCOM using either the Operations Manager administration console, PowerShell, or you can use Visual Studio itself to deploy the management pack (I will write another blog post on doing this soon).

 

Testing

In order for GPO Monitoring to occur, you need to configure Audit Policies.  This process is described in various TechNet articles, but I found the  AD DS Auditing Step-by-Step Guide particularly helpful.  Once Directory Service changes have been configured:

1. Make a change to a GPO Setting.  In my testing I changed the “Devices: Restrict CD-ROM access to locally logged-on user only” policy under the Default Domain Controllers Policy:

image

 

2. Make a change to the policy by defining it as either Enabled or Disabled.  This will generate 4 new events with Event ID 5136 in the Domain Controller’s Security event log:

image

 

The event description includes the name of the domain, the name of the user making the change, and the GUID of the GPO which was modified. 

3. Check the Active Alerts view in the Operations Manager Admin Console under the Monitoring workspace.  In this particular instance, because we had 4 new events with ID 5136, we get 4 alerts in the Active Alerts view:

image

As evidenced in the above screenshot, we receive alerts in SCOM about a change to a GPO.  The PowerShell script allowed us to translate the GUID in the event into an actual name of a GPO so that we can better track what was changed.

 

Conclusion

Although there are other samples out there illustrating how to monitor GPO changes, they used a command-line channel calling a PowerShell script to tie together two alert generating rules.  We can bypass the notifications engine, and instead use only one rule by using a custom composite Data Source module which ties the event log reader with our PowerShell script probe.    This approach could be taken to solve a great many problems when we need to take data from an event log, and use it to acquire more information using a script.  In addition, Visual Studio 2015 and the Visual Studio Authoring Extensions for Operations Manager 2012 can be versatile and powerful tools in our arsenal to implement monitoring solutions.

 

Acknowledgements

This post would not have been possible without the initial ideas from Jan Varšavský and Nathan OlmsteadKevin Holman and Brian Wren’s articles also heavily influenced my approach here, and Kevin actually helped me spot a problem with the management pack which had me stumped.  Finally I would like to acknowledge the help and input from Tyson Paul, always a fountain of great ideas.  I extend my thanks to all of them.  If you find this post useful it is due to their brilliance.  Conversely, if you have any problems with it, they are entirely of my own making.

Error installing SCOM 2012 Admin Console

Recently while working with a customer, the need arose to install the SCOM Admin console on a jump server used by some SQL Administrators. This particular customer is running SCOM 2012 RTM, without any service packs or cumulative updates/rollups. The installation of the SCOM 2012 Administration console, which is normally a pretty easy and smooth operation failed pretty quickly. Looking at the setup log, I found the following:

 

[10:05:09]: Debug: :LaunchMsi: Turning off the internal UI for C:UsersAndresDesktopSystemCenterOpsManager2012SetupAMD64ConsoleOMConsole.msi.

[10:05:09]: Debug: :LaunchMSI: Enable logging for the MSI at C:UsersAndres.daAppDataLocalSCOMLogsOMConsole.log.

[10:05:09]: Debug: :LaunchMsi: MSI C:UsersAndresDesktopSystemCenterOpsManager2012SetupAMD64ConsoleOMConsole.msi is not in silent mode. Setting the external UI.

[10:05:09]: Always: :LaunchMsi: Launching C:UsersAndresDesktopSystemCenterOpsManager2012SetupAMD64ConsoleOMConsole.msi

[10:05:09]: Always: :MsiInstallProduct finished for msi C:UsersAndresDesktopSystemCenterOpsManager2012SetupAMD64ConsoleOMConsole.msi.

[10:05:09]: Always: :LaunchMSI: Setting rollback to true

[10:05:09]: Error: :LaunchMSI: MSI C:UsersAndresDesktopSystemCenterOpsManager2012SetupAMD64ConsoleOMConsole.msi returned error 1603

[10:05:09]: Error: :ProcessInstalls: Install Item Operations console failed to install. We did not launch the post process delegate.

 

Researching the error did not turn up much, so I examined the rest of the server. I found that somehow this machine had the Microsoft Monitoring Agent installed and listed in Add/Remove Programs. This was puzzling because the name of the agent indicates SCOM 2012 R2 or better, when the agent's service name was changed. The puzzling part is that my customer's Operations Manager deployment was only 2012 RTM, with no updates or service packs. We uninstalled the SCOM 2012 Microsoft Monitoring Agent component, and then kicked off the installation of the Operations Manager Console, and this time it worked like a charm.

Going back to the log file I did see an entry of:

 

[09:53:48]: Debug: :CheckIfOperationsManagerIsInstalled: Agent found

[09:53:48]: Error: :IsValidToInstall: Agent found on this box. We cannot install

 

However, the installation did not halt because of the entries above. The only thing that happened was that the Operations Manager Management Server role was grayed out during the setup wizard. This was not a big deal for me, since we were not looking to install a Management Server. Nevertheless, the installation did not halt despite the entries above, and on the contrary, it attempted to install the Management Console component, then failed.

When the BizTalk Management Pack for Operations Manager does… well, nothing.

The following article applies to the BizTalk 2009, BizTalk 2010 and BizTalk 2013 management packs, as well as SCOM 2007 R2, SCOM 2012 and SCOM 2012 R2. In the interest of generically referring to all versions of the management pack, and of Operations Manager, I will refer to the management pack as the BizTalk management pack regardless of version, and to Operations Manager as SCOM or OM regardless of version, since all information here applies to all versions mentioned above.

 

So you have imported the BizTalk management pack, and are eagerly awaiting the results when… well, nothing happens. When you go to the various BizTalk views, nothing shows up. No BizTalk groups were discovered, no send ports, receive locations, orchestrations, hosts or host instances show up at all. What is going on?

Fortunately I have had a great deal of experience with this management pack, and with both Operations Manager and BizTalk. In my many travels I have learned that issues with the discovery of BizTalk artifacts generally fall into one of three categories, listed here in order of likelihood:

 

  1. Problems with Run As Accounts
  2. Issues with the Agent Proxy setting
  3. Discovery execution machine.

Let's take a look at each of the three causes, and how to properly resolve them so as to ensure the BizTalk discoveries run, and run successfully.

Note: For the purposes of this discussion we will assume that all BizTalk servers in the group, and all servers hosting BizTalk components have been provisioned with appropriate SCOM agents to monitor the servers. The BizTalk Management Pack does not support agentless monitoring, so an agent installed on every role server is required.

 

Problems with Run-As Accounts

 

Operations Manager monitors computers through the use of a local monitoring service known as the SCOM Agent or the Health Service. That service runs under the identity of the Default Run-As Account, which is typically Local System. The Default Run-As Account, whatever it is, probably has no privileges in BizTalk, so when SCOM tries to run discovery workflow against the BizTalk databases, and servers, the discovery will fail to get the information it is looking for. The Default Run-As Account, however, does have all the necessary rights and privileges to monitor the operating system, and the core components of the server. So what we need to do is to provide SCOM with a different set of credentials to use whenever executing discoveries or monitoring of BizTalk components.

The first step in doing so is to identify or create an account which holds all the necessary privileges for the monitoring of BizTalk Server. I like to create a brand new account, rather than using an existing service account, so as to keep separation between actions SCOM is performing and actions performed by say a BizTalk Admin. To that end, we go to Active Directory Users and Computers, and provision a new domain service account. For the purposes of this discussion let's call it the BizTalkMonitoringUser, as seen here:

BizTalk monitoring requires very specific privileges, so the next step is to make sure the account we created is member of the right groups. The BizTalkMonitoringUser account we created must:

  • Be a member of each monitored server's local Built-in Administrators group
  • Be a member of the SSO Administrators group
  • Be a member of the SSO Affiliate Administrators group
  • Be a member of the BizTalk Administrators group
  • Be a member of the BizTalk Application groups
  • Be a member of the BizTalk Operators group
  • Have the SysAdmin role on the SQL instance or instances where BizTalk databases and jobs are located.

Here is how my BizTalkMonitoringUser is provisioned:

And on the SQL Server we have:

 

Once the account has these privileges, we need to go tell Operations Manager about the account, including the user name and password, the distribution of the credentials, and which workflows the account should be associated with. The management pack guide has a section on how to do this, but some visual help might be better.

In the Operations Manager admin console go to the Administration workspace, and click on the Accounts node under Run As Configuration. Right click on Accounts and select Create Run As Account. From the Run As Account Type drop down make sure Windows is selected, as the account we are using is a Windows account. Give the account a name, and a description, and your configuration should look something like this:

Click the Next button and type in the user name and password, making sure to configure the right domain option:

Click the Next button to navigate to the Distribution Security screen. Here we tell Operations Manager how "loose" to be with the credentials we give it. Less secure will distribute the credentials to all managed computers, while More secure will send the credentials only to the servers the administrator selects. The More secure option may be a little bit more work, but it is well worth the effort, so let's choose More secure, and click Create.

At this point SCOM know about the account, but we must arrange for its proper distribution and then associate it with the workflows in the BizTalk Management Pack. To arrange the distribution, just right-click on the Run-As account you just created and select Properties as shown below.

From the properties dialog box, click on the Distribution tab, and click the Add button. In the Computer Search box, search for and find each and every server hosting BizTalk components, including all servers in the BizTalk group, servers with portal components, HTTP receive locations, SharePoint adapter servers, and the servers hosting SQL instances which have BizTalk related databases. Ensure each server is in the Selected Objects list and click OK. The list in my environment looks like this:

 

Next we need to associate the account, with SCOM workflows from the BizTalk Management Pack. To do this we have to associate the Run-As account with Run-As Profiles. When the BizTalk management pack is imported, two profiles are created out of the box under Run As Configuration -> Profiles. The two profiles created are:

We must associate the account we created with each of these two profiles. We will start by right-clicking on BizTalk Server Discovery Account, and selecting Properties, then selecting the Run As Accounts tab on the left. Click the Add button and from the Run As account dropdown box select the BizTalk Monitoring Account we created above. You may leave the radio button selected for All Targeted Objects, unless you have a need to monitor multiple BizTalk environments using different credentials (a topic for another day).

Click OK and then Save, to complete the configuration of the BizTalk Server Discovery Account Run-As Profile. Perform the same steps, to configure the same account to be used by the BizTalk Server Monitoring Account Run As Profile. When both profiles have been configured to use the Run As account with the appropriate privileges, you are done with this part of the configuration.

 

Enable Agent Proxy

Operations Manager, as it turns out, has a security constraint enabled by default. A SCOM agent running on a server is, by default, only allowed to submit information to SCOM about objects contained or owned, or hosted by the server where the agent is running. If the agent submits data (discovery or monitoring data) to a management server about objects it doesn't own then Operations Manager simply drops it.

This is a problem for some technologies. Think Active Directory, for example. Which domain controller hosts or owns or contains the domain? Clearly none of them. A domain is an entity larger than any one domain controller, and no one domain controller owns it. So for concepts such as these, we have to tell Operations Manager to disregard its normal security constraint. We have to tell Operations Manager to allow the agent on a server to act as a proxy and submit information about objects on computers other than where the agent is running.

Well, this applies to BizTalk, since no one BizTalk server owns the BizTalk group/deployment. A BizTalk Host is a group-wide execution framework, not owned by any one machine in the group. Sure, instances of a host are local to a machine, but not the host itself. The same goes for a Receive Port or Orchestration. If we do not allow the agent to submit information about objects its server doesn't own, you will see errors such as this in the Operations Manager event log:

 

Event Type: Error
Event Source: Health Service Modules
Event Category: None
Event ID: 10801Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:

     – Discovery data is stale. The discovery data is generated by an MP recently deleted.
     – Database connectivity problems or database running out of space.
     – Discovery data received is not valid.

 The following details should help to further diagnose:

 DiscoveryId: 8123404a-b439-49d8-1234-6c8c22d12345
 HealthServiceId: 4de5edba-368e-5de2-c831-d35fec214cc8
 Health service ( BTSLIT.Litware.com ) should not generate data about this managed object ( Microsoft.BizTalk.Server.2013.BAMRole ).

 

So on any and all servers hosting BizTalk artifacts we need to enable the SCOM agent to act as a proxy and allow it to submit information about objects on servers other than where it is running (even on the SQL server or servers where BizTalk databases and jobs are hosted).

To enable this functionality, open the Operations Manager console and navigate to the Administration workspace. Select the Agent Managed section on the left-side pane, and find each of the servers hosting BizTalk components. On the first one, right-click on the server, and select Properties. Click on the Security tab, and make sure the checkbox is checked for Allow this agent to act as a proxy and discover managed objects on other computers. Click OK to complete the configuration, and repeat these steps for every server hosting any BizTalk components.

 

Discovery Execution Machine

The initial discovery of BizTalk artifacts, as documented in the Management Pack Guide, only occurs on a single BizTalk server in the group. As it turns out, out of the box, the management pack looks for the first server where the group was created, and executes discoveries of artifacts there. After all, an Orchestration or Receive Location or Host is a group-wide concept, not really owned by each machine individually, and no differently configured on each machine in the group. When one creates a Send Port, the configuration for said port is the same in each machine in the group, and the port is really hosted in the Configuration database, so it makes sense to discover artifacts only on one machine in the group.

However, I have seen a couple of occurrences where the first machine to join the group is no longer in the group. I have had a couple of customers who simply shut down the old server and wheeled it away, so the group still "thinks" that machine is part of the group. And when Operations Manager goes to execute discoveries there, the machine cannot be found so they never execute.

The Management Pack Guide mentions that one can change this default configuration through an Operations Manager override. An override is a way to "supersede" the default behavior and replace it with custom behavior we desire. Unfortunately the Management Pack Guide then fails to tell us which discovery to override. Let me walk you through the correct, and complete process.

First we need to open the Operations Manager Administration Console, logged in with an account which has either Administrator or Author privileges, and navigate to the Authoring workspace. Expand the Management pack objects node, and click on Object Discoveries. If you see a yellow band across the top of the middle pane of the console, as shown below, click the "X" on the right side of the band to remove the scoping of the objects in the view.

 

Once the scoping has been removed, in the Look For text box type "BizTalk Group Discovery" as shown above, and click Find Now. This might take a moment, but soon the rest of the discoveries will be filtered out and only the discovery called BizTalk Group Discovery will be left. Right-click on BizTalk Group Discovery and select Overrides, then Override the Object Discovery and then select For all objects of class: BizTalk Run-Time Role.

Note: This last part is important as it "targets" the override to any and all servers of that class. If you have multiple BizTalk groups you are monitoring with Operations Manager, you will want to create a group containing the servers where the setting needs to be changed, excluding the ones where it does not need to change, and then target the override at the group.

The overrides properties for this discovery provide a setting for us to specify the computer where we would like discoveries to execute. Click the Override checkbox on the left-most column on the first row, with the Parameter Name of Computer Name, and then type the name of the computer you wish to execute discoveries on in the Override Value column.

Click the Edit link in the top right of the Details box, and provide a justification for the change. This is useful from an auditing perspective.

Finally select your unsealed management pack to store BizTalk customizations. Remember NOT to use the Default management pack for this, and if you don't have a management pack in which to store all BizTalk Management Pack related overrides, simply create one by clicking the New button and following the wizard. Once you have selected the management pack from the dropdown box, click OK and the change will take place.

Configuring Relative Dates for Any Scheduled SCOM Report

One of my favorite features of Operations Manager has always been the ability to tap into the data it collects and run reports. It's like being able to travel back in time, and examine what was happening on our servers 3, 6, or 12 months ago. To me, it is a type of Business Intelligence, and just like BI data, it should be leveraged for the benefit of the organization generating it. One of the ways reports are most useful, is through the scheduling functionality provided by SQL Reporting Services. OM reports can be scheduled to run on a regular basis (daily, weekly, monthly, quarterly) with convenient delivery right to your mail inbox, or a file share, and configured in pretty much any file format you would want.

A major challenge for Operations Manager administrators when scheduling reports, however, is that some of the reports which come with Operations Manager out of the box, or which ship with management packs, are not configured to use relative dates but rather use fixed dates when they are executed through the Administration Console. The ACS reports, as well as the Exchange 2010 reports are prime examples of this problem:

The above example shows the Unsuccessful Logon Attempts report from the Operations Manager Audit Reports (ACS) library. Note how the Start Date and End Date are configured as specific dates. Although this is convenient when running the report manually, because it just "runs" immediately for a two day period without having to configure a date range or other parameters, it presents a problem when the report is scheduled.

If you schedule the report above, after having run it, and configure it to run say on a weekly basis on Friday afternoons, the report which is generated on Friday afternoons will ALWAYS have the same starting date (March 12, 2014) and ending date (March 14, 2014). Always getting the same report, with the same data, for the same window of time, week after week does not make for a very useful report.

The problem is that when scheduling the report, the scheduling wizard does not give us a way to pick relative dates for the execution of the report. By relative, we mean dates relative to the time when the report is executed:

 

Notice that the Start Date, and End Date settings above are hard-wired to specific dates, which completely ruins the scheduling feature. Compare that with a report which does allow relative dates, like the Most Common Alerts report from the Microsoft Generic Report Library:

 

Notice that in this report, the From and To date fields are not hard-wired, and instead allow for the use of "Friday previous week." When the report is run, as scheduled, it figures out what that date corresponds to and adjusts it in the report execution accordingly.

 

So, does that mean that there is no hope for reports in the Audit Reports (ACS) library or other reports like the Exchange 2010 reports? Fortunately the answer is no. We can modify these reports to provide us with the relative date functionality. To do so, we must first open the reports using SQL Reporting Service's Report Builder. You can access Report Builder from the SQL Reporting Services web page by using the Report Builder button at the top:

 

I am using Report Builder 3.0 from SQL Server 2008R2, but the functionality should be the same in other versions:

 

Once Report Builder launches, we need to find and open the report we want to modify by clicking on the Open option, and then browsing to the report you wish to schedule in the Open Report dialog box.

 


 

After clicking the Open button, the report opens up in the design view of Report Builder, and it should look something like this:

 

The first thing we should do is to save a copy of the report, so we are working on the copy and not altering the original. Click the circle in the top left corner of the Report Builder window, and select Save As. Save the report with a new name, perhaps by adding some text to the end of the report file name indicating the report window. For example, if I intend to run this report for the last 7 days using relative dates, I might save my report with the name: "Access_Violation_-_Unsuccessful_Logon_Attempts_RelativeDates_Last_7_Days". With the report copy saved, we can now feel free to alter it, without fear of losing the original report, which might still be useful for manual execution in the future.

Now, note that the report data is configured using the Report Data pane on the left. If you expand the Datasets tree node, you will see the data set items which are being requested from the Operations Manager database, either the Data Warehouse or ACS databases. This particular report gets data from the ACS database, about the user, IP address, computer, logon process, authentication package, etc. The Parameters folder, meanwhile, contains the report parameters which are entered by the user at run-time. Here is where our problem is, and what we need to correct.

Start by right-clicking on the StartDate item under the Parameters folder, and select Parameter Properties. Click on the Default Values tab on the right, which shows the configuration for what are the default values used for this parameter when the report is run.

Click the "fx" button on the right side which will bring up the Expression dialog box. Here we can see how the default value for the StartDate Parameter is calculated. The formula highlighted at the top obtains Today's date, and uses the AddDays() function to add -1 days, in other words, move the date backward one day, starting today.

You should also explore the functions in the function category control, bottom left, and get familiar with the Date & Time functions. Of specific interest for a future step is the DateAdd function highlighted in the middle list control above. This function can be used to add or subtract time from a given date. The beauty of this function is that it can be used to add or subtract days, weeks, months, years, you name it. There is an excellent reference for this function, with a complete list of parameters and some sample uses, in TechNet: http://technet.microsoft.com/en-us/library/ms186819.aspx. For now, simply document the expression used for the StartDate Parameter, and do the same for the EndDate Parameter.

Armed with this knowledge, let's alter the report, so that instead of taking in input from a user. Start by right clicking on the "dataSet" item in the Datasets folder, and then click on the Parameters tab. Here we see that the dataSet gets its values for StartDate and EndDate from the Parameters we explored above.

We do not want to get the dates from the Parameters, so we will remove the link between the two. To do so, click on the "fx" button next to the StartDate, and we will have an Expression dialog box similar to the one we saw before in the Parameters Properties for StartDate, but here the expression formula is different. The expression formula here links up the dataSet's configuration to the parameters the user configures at execution. We are going to remove that link, and instead hard-wire the report to run for a fixed period of time relative to the current date. For this example we will configure the report to run for the period of the LAST 7 days, prior to today. We can use the DateAdd() function to add or subtract days as I mentioned previously. We can also use the Today() function to obtain the current date. Finally, we can next one function inside the other, so that today's date will be a parameter for the execution of the DateAdd() function, like this:

I won't get into every scheduling variation, but one other useful one is for when you want to run the report for the past month. In that scenario the value would be:

Click OK to complete the configuration of the StartDate parameter, and then configure the expression for the EndDate as follows:

 

After doing so, the Parameters tab of the Dataset properties looks like this:

 

Now we can delete the two parameters of the report, StartDate and EndDate, under the Parameters folder. Since we hard-wired the StartDate and EndDate values, the parameters are not needed. Simply right-click on each one, and select the Delete option:

 

 

Save the report, and then run it. If you schedule it from the SCOM admin console, you will notice that it does not prompt for StartDate and EndDate parameters any longer, and now when scheduled will use the relative dates you configured.