Recently, I had an opportunity to produce a proof of concept around leveraging Operations Management Suite (OMS) as a centralized point of monitoring, analytics, and alerting. As in any good Azure project, codifying your Azure resources in an Azure Resource Manager template should be table stakes. While the official ARM schema doesn't yet have all the children for the Microsoft.OperationalInsights/workspaces type, the Log Analytics documentation does document samples for interesting use cases that demonstrate data sources and saved searches. Notably absent from this list, however, is anything about the /schedules and /action types. Now, a clever Azure Developer will realize that an ARM template is basically an orchestration and expression wrapper around the REST APIs. In fact, I've yet to see a case where an ARM template supports a type that's not provided by those APIs. With that knowledge and the alert API samples, we can adapt the armclient put <url> examples to an ARM template.
The Resources
What you see as an alert in the OMS portal is actually 2 to 3 distinct resources in the ARM model:
A mandatory schedules type, which defines the period and time window for the alert
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Search-Id')]"
],
"properties": {
"Interval": 5,
"QueryTimeSpan": 5,
"Active": "true"
}
}
A mandatory actions type, defining the trigger and email notification settings
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Alert-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Schedule-Id')]"
],
"properties": {
"Type": "Alert",
"Name": "[parameters('Search-DisplayName')]",
"Threshold": {
"Operator": "gt",
"Value": 0
},
"Version": 1
}
}
An optional, secondary, actions type, defining the alert's webhook settings
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Action-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Schedule-Id')]"
],
"properties": {
"Type": "Webhook",
"Name": "[variables('Action-Name')]",
"WebhookUri": "[parameters('Action-Uri')]",
"CustomPayload": "[parameters('Action-Payload')]",
"Version": 1
}
}
The Caveats
The full template adds a few eccentricities that are necessary for fully functioning resources.
First, let's talk about the search. At the OMS interface, when you create a search, it is named <Category>|<Name> behind the scenes. It is also lower-cased. Now, the APIs will not downcase any name you send in, but if you use the interface to update that API-deployed search, you will duplicate it rather than being prompted to overwrite. This tells us that for whatever reason, OMS' identifiers are case-sensitive. This is why the template accepts a search category and name in parameters, then use variables to lower them and match the interface's naming convention.
Second, a schedule's name must be globally unique to your workspace, though the actions need not be. Again, the OMS UI will take over here and generate GUIDs for the resources behind the scenes. Unfortunately for us, ARM Templates don't support random number or GUID generation - a necessary evil of being deterministic. While the template could accept a GUID by parameter, I find that asking my users for GUIDs is impolite - a uniquestring() function on the already-unique search name should suffice for most use cases. Since the actions don't require uniqueness, static names will do.
Lastly, this template is not idempotent. Idempotency is a characteristic of a desired state technology that focuses on ensuring the state (in our case, the template), rather than prescribing process (create this, update that, delete the other thing). Each of Azure's types is responsible for ensuring that they implement idempotency as much as possible. For some resources like the database extensions type, which imports .bacpac files, the entire type opts out of idempotency (once data is in the database, no more importing). For others (such as Azure VMs), only changes in certain properties like administrator name/password break idempotency. In the OMS search and schedule's case, if either one exists when you deploy the template, the deployment will fail with a code of BadRequest. To add insult to injury, deleting the saved search from the interface does not delete the schedule. Neither does deleting the alert from the UI - it simply flips the scheduled to "Enabled": false. There is, in fact, no way to delete the schedule via the UI - you must do so via the REST API, similar to the following:
armclient delete /{Schedule ResourceID}?api-version=2015-03-20
This means that, for now, any versioned solution which desires to deploy ARM template updates to its searches and alerts will need to either:
- Delete the existing search/schedule, then redeploy OR
- Deploy the updated version side-by-side (named differently), then clean up the prior version's items.
Luckily ours was a POC, so deleting and re-creating was very much an option. I look forward to days when search and alerting types become first-class, idempotent citizens. Until then, I hope this approach proves useful!
The Template
The template below deploys a search, a schedule, a threshold action (the alert), and a webhook action (the action) that includes a custom JSON payload (remember to escape your JSON!). With a little work, this template could be reworked to segregate the alert-specific components from the search so that multiple alerts could be deployed to the same search or adapted for use with Azure Automation Runbooks. Enjoy!
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"Action-Payload": {
"type": "string"
},
"Action-Uri": {
"type": "string"
},
"Alert-DisplayName": {
"type": "string"
},
"Search-Category": {
"type": "string"
},
"Search-DisplayName": {
"type": "string"
},
"Search-Query": {
"type": "string"
},
"Workspace-Name": {
"type": "string"
},
"Workspace-Location": {
"type": "string"
}
},
"variables": {
"Action-Name": "webhook",
"Alert-Name": "alert",
"Schedule-Name": "[uniquestring(variables('Search-Name'))]",
"Search-Name": "[toLower(concat(variables('Search-Category'), '|', parameters('Search-DisplayName')))]",
"Search-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches', parameters('Workspace-Name'), variables('Search-Name'))]",
"Schedule-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'))]",
"Alert-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'), variables('Alert-Name'))]",
"Action-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'), variables('Action-Name'))]"
},
"resources": [
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [],
"properties": {
"Category": "[parameters('Search-Category')]",
"DisplayName": "[parameters('Search-DisplayName')]",
"Query": "[parameters('Search-Query')]",
"Version": "1"
}
},
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Search-Id')]"
],
"properties": {
"Interval": 5,
"QueryTimeSpan": 5,
"Active": "true"
}
},
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Alert-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Schedule-Id')]"
],
"properties": {
"Type": "Alert",
"Name": "[parameters('Search-DisplayName')]",
"Threshold": {
"Operator": "gt",
"Value": 0
},
"Version": 1
}
},
{
"name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Action-Name'))]",
"type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
"apiVersion": "2015-03-20",
"location": "[parameters('Workspace-Location')]",
"dependsOn": [
"[variables('Schedule-Id')]"
],
"properties": {
"Type": "Webhook",
"Name": "[variables('Action-Name')]",
"WebhookUri": "[parameters('Action-Uri')]",
"CustomPayload": "[parameters('Action-Payload')]",
"Version": 1
}
}
],
"outputs": {}
}
Windows Server 2016 RTM and System Center 2016 RTM are now live now generally available through the usual sources, including Technet and MSDN. This coincides with the release of a Server 2016 RTM IaaS offering in Azure which is now open to all subscriptions.