Azure Monitor Container Insights cost presets (preview)

Microsoft have recently added a new preview feature within the Azure portal to perform some cost optimization on the Container Insights data that is collected to a Log Analytics workspace. This is helpful as going with the default collection could leave you with unnecessary cost, as you don’t care about particular namespaces, or you’ve got a non-production cluster, so don’t care about collecting data every minute, but still want to monitor your cluster and workloads.

Here’s how you can quickly set this up.

• From the Azure Portal, open up the blade for your Kubernetes resource (works for both AKS and Arc enabled Kubernetes

• Navigate to Monitoring / Insights and then Configure azure monitor (If you’ve not previously enabled it)

If you have configure container insights previously, click on Monitor Settings :

From the blade that’s opened, you’ll need to make sure that Use Managed Identity (preview) is enabled. This appears for Azure Arc for Kubernetes enabled clusters. If this isn’t checked, you will find the Costs presets (preview) list box is grayed out.

From the list box, you can choose from the following profiles:

• Standard

• Cost-optimized

• Custom

• None

Standard profile has the following settings:

• 1 minute collection frequency

• No namespace filtering

• Syslog collection disabled

Cost-optimized profile has the following settings:

• 5 minute collection frequency

• 3 namespaces excluded [kube-system, gatekeeper-system, azure-arc]

• Syslog collection disabled

You can use the defined standards as the basis for your own custom collection profile.

Select a profile from the drop down and click on Edit collection settings.

You can now modify the configuration to meet your requirements. For ContainerLog filtering, you can use the link here for more information. The profile will be saved as Custom within the Cost presets list box.

You can also enable Syslog collection for security events on your nodes, but that will increase your Log Analytic costs somewhat, depending on how busy your cluster is!

Azure Arc delivers SQL Databases as Resources in Azure

The SQL extension for Arc-connected servers itself isn’t new, however, it has had a new feature added that is currently in preview; which is the ability to create Azure resources that represent your SQL Database through the Arc-connected Servers.

You can see this will detect objects for SQL instances, named SQL instances, and the databases themselves become objects in Azure

If you already have this installed this is one configuration setting you may want to check and that is “Has NT AUTHORITY\SYSTEM in the sysadmin role.” This is listed in the prerequisites of the install instructions that can be found here View SQL Server databases - SQL Server | Microsoft Learn

It is worth noting this currently works only for the Microsoft.HybridCompute type and not for example the VMware connected Arc systems Microsoft.ConnectedVMwarevSphere.

There are some properties here listed as unknown however, this is a bug that has already been fixed.

This is the next step to bringing SQL databases into the Azure control plane, creating alerts on recovery models, encryption state, or perhaps statistic creation. Enabling more visibility and control into the Hybrid Cloud journey.

This is the first public preview of this feature and I’ll definitely be keeping an eye out for future revisions and enhancements.

Azure Arc Connected vSphere (Preview)

This isn’t a groundbreaking demo in terms of the end result; we deploy some VMs, however, the interesting conversation is about how we got there and what we can do with these VMs. With Azure Arc, we can now connect vSphere to the Azure platform, meaning using Azure we can use cloud methods to deploy servers such as Azure templates. This is a step towards creating an environment with a single-pane-of-glass, at least to view and query your IT estate.

The features today are simple, but Azure is a constantly evolving platform and when you are connected, as new features and services are developed and released, and the ecosystem evolves, you can grow with it.

The goal of this article is to give you some ideas about why connecting your vSphere cluster to Azure via Azure Arc and how having VMWare-connected VMs can start providing a path to a consistent cloud experience. This allows administrators to develop management processes in the cloud without requiring a complete lift and shift of all existing servers. As of this writing, this feature is still in preview.

View fullsize

This is assuming you have deployed the Azure Arc bridge to connect your on-premises resources. This process is evolving and is not meant to be the focus of this article.

Once connected using the Azure portal, you can do a simple VM deployment.

Click, click, VM name, deploy-demo-01 next, next, finish.

View fullsize

We can save this as an ARM template, I created a template spec and saved it to Azure, which I am using through the portal to deploy VM deploy-demo-02 but you could use pipeline tools such as Azure DevOps or other CI/CD tooling or templates from PowerShell command line.

View fullsize

You could use PowerShell to deploy that ARM Template to deploy deploy-demo-03

View fullsize

View fullsize

you could convert this into a Bicep template if that’s your standard.

View fullsize

What I do have now is 3 new vSphere hosted VMs all deployed via Azure ARM

View fullsize

however, I never touched the VMware console, and I now have an Azure Resource object.

You have access to basic controls. You could assign this via Azure RBAC rather than having administrators and users needing to access the vSphere console. There are options for resizing, adding disks, and managing networking, standardizing the experience.

View fullsize

In addition, you can also onboard existing deployed vSphere VMs and enable guest management to help manage things like VMware tools versions from Azure.

While the monitoring agent options are changing, I’ll keep this simple. You could also include VM extension such as the monitoring agent in your template.

which in term allows us to query these servers via Log analytics. This means it can be connected to alerts like low disk space and other services like Azure Policy.

View fullsize

This space is changing, there are a lot of new features coming to Azure Arc which we see as a gateway to connecting your existing IT servers to Azure allowing companies to gain the benefits of Azure without having to execute migrations for every server.

Hopefully, you found this insightful or informative

Azure Arc Connected K8s Billing usage workbook error - 'union' operator: Failed to resolve table expression named 'ContainerLogV2'

Using ‘Container Insights’ for Kubernetes can get expensive, I recently saw a customer creating 19TB of logs a month from logging container insights from an AKS cluster.

You may want to check what logs you are capturing as suggested through Microsoft guides Monitoring cost for Container insights - Azure Monitor | Microsoft Docs which suggests one of the first steps to check billing usage.

As of writing this if you enable container insights and try to view Data Usage you get an error. ‘union' operator: Failed to resolve table expression named 'ContainerLogV2’. ContainerLogV2 is in preview or if you are just aren’t ready to deploy that, here is how to fix the report manually. I also have included the Full report at the end.

View fullsize

However, depending on when you deploy this and how you deployed insight metrics you may hit this error.

As of writing this, ContainerLogV2 is in preview and you can follow this guide to deploy it. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs. If you want to gain some quick insight into your usage you can edit the workbook. We just need to get through the maze of edits to get to the underlying query.

• Select ‘Azure Arc’ services

• select ‘Kubernetes clusters’ under infrastructure

• select your K8s cluster

• select ‘Workbooks’ under Monitoring

• select ‘Container Insights Usage’ workbook

• you can either edit this one or ‘Save As’ and create a new name. I will create a copy.

View fullsize

Open your chosen workbook

you can now edit the group item.

by Namespace

Editing the Namespace selection Dropdown parameter query

Hopefully, you get the gist of the process. There are a few more queries to edit.

And

This will now work for Data Usage. This is likely a temporary solution until ContainerLogV2 comes out of preview. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs

Gallery Template code

{
  "version": "Notebook/1.0",
  "items": [
    {
      "type": 9,
      "content": {
        "version": "KqlParameterItem/1.0",
        "crossComponentResources": [
          "{resource}"
        ],
        "parameters": [
          {
            "id": "670aac26-0ffe-4f29-81c2-d48911bc64b6",
            "version": "KqlParameterItem/1.0",
            "name": "timeRange",
            "label": "Time Range",
            "type": 4,
            "description": "Select time-range for data selection",
            "isRequired": true,
            "value": {
              "durationMs": 21600000
            },
            "typeSettings": {
              "selectableValues": [
                {
                  "durationMs": 300000
                },
                {
                  "durationMs": 900000
                },
                {
                  "durationMs": 3600000
                },
                {
                  "durationMs": 14400000
                },
                {
                  "durationMs": 43200000
                },
                {
                  "durationMs": 86400000
                },
                {
                  "durationMs": 172800000
                },
                {
                  "durationMs": 259200000
                },
                {
                  "durationMs": 604800000
                },
                {
                  "durationMs": 1209600000
                },
                {
                  "durationMs": 2419200000
                },
                {
                  "durationMs": 2592000000
                },
                {
                  "durationMs": 5184000000
                },
                {
                  "durationMs": 7776000000
                }
              ],
              "allowCustom": true
            }
          },
          {
            "id": "bfc96857-81df-4f0d-b958-81f96d28ddeb",
            "version": "KqlParameterItem/1.0",
            "name": "resource",
            "type": 5,
            "isRequired": true,
            "isHiddenWhenLocked": true,
            "typeSettings": {
              "additionalResourceOptions": [
                "value::1"
              ],
              "showDefault": false
            },
            "timeContext": {
              "durationMs": 21600000
            },
            "timeContextFromParameter": "timeRange",
            "defaultValue": "value::1"
          },
          {
            "id": "8d48ec94-fde6-487c-98bf-f1295f5d8b81",
            "version": "KqlParameterItem/1.0",
            "name": "resourceType",
            "type": 7,
            "description": "Resource type of resource",
            "isRequired": true,
            "query": "{\"version\":\"1.0.0\",\"content\":\"\\\"{resource:resourcetype}\\\"\",\"transformers\":null}",
            "isHiddenWhenLocked": true,
            "typeSettings": {
              "additionalResourceOptions": [
                "value::1"
              ],
              "showDefault": false
            },
            "timeContext": {
              "durationMs": 21600000
            },
            "timeContextFromParameter": "timeRange",
            "defaultValue": "value::1",
            "queryType": 8
          },
          {
            "id": "1b826776-ab99-45ab-86db-cced05e8b36d",
            "version": "KqlParameterItem/1.0",
            "name": "clusterId",
            "type": 1,
            "description": "Used to identify the cluster name when the cluster type is AKS Engine",
            "isHiddenWhenLocked": true,
            "timeContext": {
              "durationMs": 0
            },
            "timeContextFromParameter": "timeRange"
          },
          {
            "id": "67227c35-eab8-4518-9212-1c3c3d564b20",
            "version": "KqlParameterItem/1.0",
            "name": "masterNodeExists",
            "type": 1,
            "query": "let MissingTable = view () { print isMissing=1 };\r\nlet masterNodeExists = toscalar(\r\nunion isfuzzy=true MissingTable, (\r\nAzureDiagnostics \r\n| getschema \r\n| summarize c=count() \r\n| project isMissing=iff(c > 0, 0, 1)\r\n) \r\n| top 1 by isMissing asc\r\n);\r\nprint(iif(masterNodeExists == 0, 'yes', 'no'))\r\n",
            "crossComponentResources": [
              "{resource}"
            ],
            "isHiddenWhenLocked": true,
            "timeContext": {
              "durationMs": 0
            },
            "timeContextFromParameter": "timeRange",
            "queryType": 0,
            "resourceType": "{resourceType}"
          }
        ],
        "style": "pills",
        "queryType": 0,
        "resourceType": "{resourceType}"
      },
      "name": "pills"
    },
    {
      "type": 1,
      "content": {
        "json": "Please note that the Container Insights Usage workbook for AKS Engine clusters shows your billable data usage for your cluster's entire workspace ({resource:name}), and not just the cluster itself ({clusterId}). ",
        "style": "info"
      },
      "conditionalVisibility": {
        "parameterName": "resourceType",
        "comparison": "isEqualTo",
        "value": "microsoft.operationalinsights/workspaces"
      },
      "name": "aks-engine-billable-data-shown-applies-to-entire-workspace-not-just-the-cluster-info-message",
      "styleSettings": {
        "showBorder": true
      }
    },
    {
      "type": 11,
      "content": {
        "version": "LinkItem/1.0",
        "style": "tabs",
        "links": [
          {
            "id": "3b7d39f2-38c5-4586-a155-71e28e333020",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "Overview",
            "subTarget": "overview",
            "style": "link"
          },
          {
            "id": "7163b764-7ab2-48b3-b417-41af3eea7ef0",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Table",
            "subTarget": "table",
            "style": "link"
          },
          {
            "id": "960bbeea-357f-4c07-bdd9-6f3f123d56fd",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Namespace",
            "subTarget": "namespace",
            "style": "link"
          },
          {
            "id": "7fbcc5bd-62d6-4aeb-b55e-7b1fab65716a",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Log Source",
            "subTarget": "logSource",
            "style": "link"
          },
          {
            "id": "3434a63b-f3d5-4c11-9620-1e731983041c",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Diagnostic Master Node ",
            "subTarget": "diagnosticMasterNode",
            "style": "link"
          }
        ]
      },
      "name": "tabs"
    },
    {
      "type": 1,
      "content": {
        "json": "Due to querying limitation on Basic Logs, if you are using Basic Logs, this workbook provides partial data on your container log usage.",
        "style": "info"
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "overview"
      },
      "name": "basic-logs-info-text"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "title": "Container Insights Billing Usage",
        "items": [
          {
            "type": 1,
            "content": {
              "json": "<br/>\r\nThis workbook provides you a summary and source for billable data collected by [Container Insights solution.](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview)\r\n\r\nThe best way to understand how Container Insights ingests data in Log Analytics workspace is from this [article.](https://medium.com/microsoftazure/azure-monitor-for-containers-optimizing-data-collection-settings-for-cost-ce6f848aca32)\r\n\r\nIn this workbook you can:\r\n\r\n* View billable data ingested by **solution**.\r\n* View billable data ingested by **Container logs (application logs)**\r\n* View billable container logs data ingested segregated by **Kubernetes namespace**\r\n* View billable container logs data ingested segregated by **Cluster name**\r\n* View billable container log data ingested by **logsource entry**\r\n* View billable diagnostic data ingested by **diagnostic master node logs**\r\n\r\nYou can fine tune and control logging by turning off logging on the above mentioned vectors. [Learn how to fine-tune logging](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config)\r\n\r\nYou can control your master node's logs by updating diagnostic settings. [Learn how to update diagnostic settings](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "name": "workbooks-explanation-text"
          },
          {
            "type": 1,
            "content": {
              "json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "no"
            },
            "name": "master-node-logs-are-not-enabled-msg"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "overview"
      },
      "name": "workbook-explanation"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "union withsource = SourceTable AzureDiagnostics, AzureActivity, AzureMetrics, ContainerLog, Perf, KubePodInventory, ContainerInventory, InsightsMetrics, KubeEvents, KubeServices, KubeNodeInventory, ContainerNodeInventory, KubeMonAgentEvents, ContainerServiceLog, Heartbeat, KubeHealth, ContainerImageInventory\r\n| where _IsBillable == true\r\n| project _BilledSize, TimeGenerated, SourceTable\r\n| summarize BillableDataBytes = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), SourceTable\r\n| render piechart\r\n\r\n\r\n",
              "size": 3,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Data from Container Insights",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "xAxis": "TimeGenerated",
                "createOtherGroup": 100,
                "seriesLabelSettings": [
                  {
                    "seriesName": "LogManagement",
                    "label": "LogManagementSolution(GB)"
                  },
                  {
                    "seriesName": "ContainerInsights",
                    "label": "ContainerInsightsSolution(GB)"
                  }
                ],
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "50",
            "showPin": true,
            "name": "billable-data-from-ci"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "table"
      },
      "name": "by-datatable-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| summarize Total=sum(BillableDataBytes) by Namespace\r\n| render piechart\r\n",
              "size": 3,
              "showAnalytics": true,
              "title": "Billable Container Log Data Per Namespace",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "tileSettings": {
                "showBorder": false,
                "titleContent": {
                  "columnMatch": "Namespace",
                  "formatter": 1
                },
                "leftContent": {
                  "columnMatch": "Total",
                  "formatter": 12,
                  "formatOptions": {
                    "palette": "auto"
                  },
                  "numberFormat": {
                    "unit": 17,
                    "options": {
                      "maximumSignificantDigits": 3,
                      "maximumFractionDigits": 2
                    }
                  }
                }
              },
              "graphSettings": {
                "type": 0,
                "topContent": {
                  "columnMatch": "Namespace",
                  "formatter": 1
                },
                "centerContent": {
                  "columnMatch": "Total",
                  "formatter": 1,
                  "numberFormat": {
                    "unit": 17,
                    "options": {
                      "maximumSignificantDigits": 3,
                      "maximumFractionDigits": 2
                    }
                  }
                }
              },
              "chartSettings": {
                "createOtherGroup": 100,
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "name": "billable-data-per-namespace"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "namespace"
      },
      "name": "by-namespace-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 9,
            "content": {
              "version": "KqlParameterItem/1.0",
              "crossComponentResources": [
                "{resource}"
              ],
              "parameters": [
                {
                  "id": "d5701caa-0486-4e6f-adad-6fa5b8496a7d",
                  "version": "KqlParameterItem/1.0",
                  "name": "namespace",
                  "label": "Namespace",
                  "type": 2,
                  "description": "Filter data by namespace",
                  "isRequired": true,
                  "query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| project ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| distinct Namespace\r\n| project value = Namespace, label = Namespace, selected = false\r\n| sort by label asc",
                  "crossComponentResources": [
                    "{resource}"
                  ],
                  "typeSettings": {
                    "additionalResourceOptions": [
                      "value::1"
                    ],
                    "showDefault": false
                  },
                  "timeContext": {
                    "durationMs": 0
                  },
                  "timeContextFromParameter": "timeRange",
                  "defaultValue": "value::1",
                  "queryType": 0,
                  "resourceType": "{resourceType}"
                }
              ],
              "style": "pills",
              "queryType": 0,
              "resourceType": "{resourceType}"
            },
            "name": "namespace-pill"
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| join hint.strategy=shuffle (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by LogEntrySource, ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n)\r\n| extend sourceNamespace = strcat(LogEntrySource, \"/\", Namespace)\r\n| summarize Total=sum(BillableDataBytes) by sourceNamespace\r\n| render piechart",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Data Per Log-Source Type for Namespace Selected",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "createOtherGroup": 100,
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "35",
            "showPin": true,
            "name": "billable-data-per-log-source",
            "styleSettings": {
              "showBorder": true
            }
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n| join hint.strategy=shuffle ( \r\nContainerLog\r\n| project _BilledSize, ContainerID, TimeGenerated, LogEntrySource\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n)\r\n| summarize ContainerLogData = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), LogEntrySource\r\n| render linechart\r\n\r\n",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Container Log Data for Namespace Selected",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "xSettings": {},
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "65",
            "conditionalVisibility": {
              "parameterName": "namespace:label",
              "comparison": "isNotEqualTo",
              "value": "<unset>"
            },
            "showPin": true,
            "name": "billable-container-log-data-for-namespace-selected",
            "styleSettings": {
              "showBorder": true
            }
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "logSource"
      },
      "name": "by-log-source-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 1,
            "content": {
              "json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "no"
            },
            "name": "master-node-logs-are-not-enabled-msg-under-masternode-tab"
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "AzureDiagnostics\r\n| summarize Total=sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), Category\r\n| render barchart\r\n\r\n\r\n",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Diagnostic Master Node Log Data",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "ySettings": {
                  "unit": 2,
                  "min": null,
                  "max": null
                }
              }
            },
            "customWidth": "75",
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "yes"
            },
            "showPin": true,
            "name": "billable-data-from-diagnostic-master-node-logs"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "diagnosticMasterNode"
      },
      "name": "by-diagnostic-master-node-tab"
    }
  ],
  "fallbackResourceIds": [
    "/subscriptions/357e7522-d1ad-433c-9545-7d8b8d72d61a/resourceGroups/ArcResources/providers/Microsoft.Kubernetes/connectedClusters/sv5-k8sdemo"
  ],
  "fromTemplateId": "community-Workbooks/AKS/Billing Usage Divided",
  "$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json"
}