For a clearer picture

Category: Azure

Part 1 – Azure Monitoring for free

Part 1 – Azure Monitoring for free

Yes; free. No gimmicks. No dodgy sales calls. No cracked license keys. Totally free Azure monitoring from Microsoft themselves.

OK. Let us start off by agreeing what we mean by free. When you buy a car you expect to get the wheels, the steering wheel and the brakes as well as many other components as part of the overall cost of the car. Imagine going to drive away from the forecourt just as the dealer takes the wheels off and tells you they cost extra. That wouldn’t be cricket. And it wouldn’t be much of a car either.

Same idea with Azure. When you “buy” a resource; you get some monitoring functionality as part of the cost of the resource. That is what I mean by free. Although the grammar guardians mightwell prefer me to say “at no additional cost” it doesn’t sound so catchy as a title.

At Ignite 2018 Microsoft announced a re-imagining of the Azure Portal and they did a great job of it. What had become a slightly confusing collection of monitoring options suddenly had a much more intuitive layout.

But what might not be entirely clear from the portal is what is free and what you really do need to pay for.

That is where this series of articles comes in. To ensure that you don’t accidentally enable functionality that incur costs that neither you, nor your bosses, expect.

Hopefully it is good news that as soon as you start using Azure, Azure Monitor starts recording what activities are taking place against the resource and providing a base level of performance metrics for that resource. Note – this is the Cloud. Everything is subject to change at very little notice and, when it involves increasing prices, in incredibly small print on obscure pages. This series is correct of October 5th, 2018. Please let me know if you spot something is out of date and I’ll amend it. Thanks.

So we’ll be looking at:

  • Service Health – which highlights problems with Azure Services that impact you now
  • Activity Logs – which record when resources are created or modified
  • Metrics – which tell you how the resource is performing and the resources that it’s consuming.
  • Lessons Learned.

Part 5 – Azure MP – Lessons learned

Part 5 – Azure MP – Lessons learned

I have to admit I’m not a fan.

The organisations I have worked with need version and source code control. They insist on authoring SCOM management packs via Visual Studio. They require the authoring to be peer reviewed and source code to be stored appropriately and deployed via a robust release management framework. Basically, SCOM management pack development is treated the same as any other development

1. Azure monitoring is template based. This means that a lot of these requirements are not met:
– I can’t code once in Visual Studio and deploy the same code to multiple environments. The best I can do is run a wizard in one environment and then export the MP and store it. I guess this could all be automated but it isn’t straight forward.

2. When you run the template wizard under authoring you can only run the wizard once for each subscription. This means that all your monitoring for a subscription has to be saved to the same management pack. And if someone makes a mistake or worse, accidentally deletes it, it impacts my whole Azure Monitoring.

3. The classes get created on the fly by the authoring template wizard; this makes it much more difficult to code against as you don’t know the class name until it has already been discovered.

4. The instances of the Azure Resources that are discovered don’t get user friendly names or descriptions and are difficult to decipher what they actually relate to; especially for non-specialists.

5. The resultant alerts are not user friendly and are difficult to integrate into ticketing systems.

SCOM isn’t a Cloud monitoring solution. The cloud is all about scaling on demand, elasticity, consuming resources as and when you need them and not a second longer.

SCOM doesn’t work that way. Discoveries run every few hours or even just once per day. That isn’t just a lifetime in Cloud terms; it is multiple generations. Monitoring virtual machines (IaaS) doesn’t require the Azure MP as they can be monitored as any other virtual machine given the usual requirements on connectivity and authentication. PaaS and SaaS monitoring requires something altogether more dynamic than SCOM.

So when moving to the Cloud; think about how you are going to monitor all of this transient, ephemeral resources whose lifecycles are beyond the scope of SCOM monitoring.

For Cloud monitoring from a Microsoft perspective; Log Analytics is the way forward and that is what I’ll concentrate on in a future series of articles.

Part 4 – Deploying the SCOM Azure Management Pack

Part 4 – Deploying the SCOM Azure Management Pack

Download details

The latest version of the Azure Management Pack is available here.

I’ll run through installing the Azure Management Pack and carrying out the configuration which will involve:

  • Configuring the connection to Azure
  • Running the Authoring Template wizard to configure the monitoring

So having downloaded the MP, read the management pack guidelines and ensured that I have the following complete:

  • Created a Resource Pool for Azure Monitoring
  • Ensured the Management Servers (or Gateway Servers) that are in the Resource Pool are added to the resource pool, have internet connectivity and .Net Framework 4.5.2 installed
Extract Management Pack

I’m ready to go. Lets extract the Management Pack by running the .msi

Accept the license agreement.

Location to extract to:

And go ….

And finish …

And we’ll see 2 Management Pack bundles that we need to import to SCOM.

Let’s import them:

Configure Management Pack Administration details

And then start the Azure configuration process. You’ll notice that on the Administration blade; there is now an Azure icon.

And then choose to add an Azure subscription. The first question you will get is how to authenticate to Azure.

A User Principal Name (UPN) is simpler to configure, but it does not work when multi-factor authentication is enforced.

A Service Principal Name (SPN) works in any environment and allows assigning permissions in a more flexible way. Using a service principal allows the management pack to be used in environments with multi-factor authentication. Ideally you should create the Service Principal Name ahead of deployment but I’m going to take the lazy route for this demonstration and let the configuration wizard create it on the fly.

More on SPNs here

I don’t of any reason why you would need to change the Azure endpoints so leave these on the defaults.

And then select auto-created SPN and enter the details requested.

Then click Add Subscription to start the configuration of the subscription:

Confirm that it has worked:

And then select the SCOM Resource Pool that will be responsible for the Azure monitoring. Note that the Management Servers or Gateway Servers in the resource pool do not need direct internet connectivity; there is the option here to configure communication via a proxy.

You’ll be shown a list of subscriptions to choose to monitor. I’ve only got the one so will just select that.

And that is the Administration configuration complete. At this moment we have no monitoring in place. We still need to run the authoring template wizard. But we have the framework in place to start monitoring.

Azure Monitoring Template Wizard

Sadly you can only run the wizard once per subscription which means that you can’t slice and dice your Azure monitoring into more granular logical management packs.

Lets’ run the wizard:

Enter a Name and Management Pack to save the configuration into:

Select the subscription and do a full Azure scan. Take note of the comments at the bottom of the window; a full scan can take 1 to 2 hours for 1000 Azure resources. After the initial full scan you can just do refreshes but I am concerned about enterprise scalability.

You then get a list of all the discovered Azure resources and can select the items that you want to monitor. I’ve just selected my Microsoft.insights resources.

  • Alert Rules – I’ll look at in the next article in this series
  • Components – this is the Azure Resource Name (top level – ping-SCOM_Prod1)
  • Web Tests – the actual URL tests themselves
    • If you want to exclude specific Azure Resources you can do so here:

      Select the data you want SCOM to collect. Be aware that the URL tests just collect the top line item “Test Duration” (which equates to response time) so even though you might select all the performance counters; you won’t be able to actually collect them.

      You can also select to alert on conditions; such as Test Duration > 1200 milliseconds.

      Once you click Finish; you’ll see a Microsoft Azure folder on the Monitoring blade and you’ll be able to see the performance metrics and the Resource State:

      Performance View

      State View

      There naming convention which includes Azure subscription id and Azure Resource Type makes it very difficult for users to easily understand what is being monitored.

      Alert View

      As you can see; we do get alerts when thresholds are exceeded but the alerts are somewhat cryptic. They are not user friendly and integrating with ticketing systems will be challenging.

Part 2 – Global Service Monitor Migration Script

Part 2 – Global Service Monitor Migration Script

It is important to migrate your GSM scripts using the migration tool as this will tag the Azure Application Insights tests in a way that ensures that you do not have to pay for them. If you manually recreate your GSM monitoring as Azure Application Insights tests then you will fall under Azure licensing charges.

I have only tested the migration script with taking basic URL web tests from SCOM to Azure Application Insights (ping tests). These are generally very basic tests on whether a URL is available along with the response time and the option of content matching and checking returned status codes.

I found the whole process very straight forward and after many tests; I have never seen it fail. I have not however tried to migrate Visual Studio Web Tests. If I find some time I’ll give those a go later.

Let’s take a look at my GSM Tests

I have 2 tests configured:

Each test has 2 distinct URL checks:

Which are monitored from 5 locations:

And I want an alert if the transaction response time is greater than 5 seconds; if the return code is greater than 400 or if the content match doesn’t contain specific text. I’m also collecting performance data. Take note of these settings as when you move to Azure Application Insights the options are much more limited.

So; let’s run the Migration Script.

First up; if you don’t have the Azure PowerShell module installed, you’ll need to get this configured. Here is the walk through.

Install-Module -Name AzureRM

I then connect to my Azure Subscription to confirm some details;

I’ve cropped them out of the screenshot but make a note of the subscription name.

And then we can run the GSM Migration Script. Best practice naming conventions for Azure Resources are listed here.

MigrateGSMToAI.ps1 -SubscriptionName "**Subscription Name**" -AzureResourceGroupName "**Resource Group Name**" -ResourceLocation "**Location**"

And you can watch the migration take place:

And then there are the log files that you can review if there are any issues. I’ve never had the script fail.

Azure Application Insights Resource

Fire up the Azure Portal and go to your Application Insights resource and you’ll see the migrated Global Service Monitor tests.

My initial concerns is that all GSM tests are migrated to a single Application Insights resource. I can’t see anyway to slice and dice this as part of the migration. However, it is possible to run the Global Service Monitor migration script multiple times and you could then edit the resulting Application Insights Resources in the Azure portal. A bit of a pain; it would be neater to be able to do this as part of the migration script.

Viewing Test Data

If you click on the Availability blade then you can see your tests in action.

You can then click on an individual test and see more detailed results.

You can also click on edit test and change the configuration:

  • Test Name – this cannot be edited so you need to get this right in SCOM before the migration.
  • Test Type – this cannot be edited; it is set by the type of Global Service Monitor test you migrated. As my GSM tests were basic URL checks (not Visual Studio Web Tests) they have been migrated as URL Ping Tests.
  • The URL – this can be edited.
  • Test Name – this cannot be edited so you need to get this right in SCOM before the migration.
  • Parse dependent requests – embedded resources such as links and images are parsed.
  • Enable retries – if the test fails then the number of times to retry before a failure is reported.
  • Test Frequency – every 5, 10 or 15 minutes.
  • Test Locations – taken from the Global Service Monitor configuration and can be edited in Azure.
  • Success Criteria – this gives a few options but a lot less than was available in Global Service Monitor. These are quite disappointing:
    • – Test time out greater than 30 seconds (with options of 60, 90 or 120 seconds). This is not the same as setting an alert on response time which can be in either in Azure alerting or via the SCOM Management Pack.
    • – HTTP response – status code =
    • – Content Match – must contain
Quick review

I noticed that the shortest test interval schedule is every 5 minutes. I think this is a lifetime in terms of web site availability but this is a like for like setting with Global Service Monitor.