"CannotChangeInventoryId is not allowed" Enabling a VMWare OS template in Azure via VMWare Arc-Connected vSphere

While working on a VMware Arc-connected instance of vSphere I had an issue enabling a template for use in Azure. I had created a template deleted it then tried to recreate it with the same name and received the following error.

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"CannotChangeInventoryId","message":"Resource 'template-Win2019STD' was previously configured with InventoryItemId 'vmtpl-vm-2015'. Changing InventoryItemId is not allowed."}]}

In the resource group and select show hidden items, we can see a template with that name

View fullsize

if we actually look at the object or JSON view we can see this is the ID from the error

View fullsize

simply delete the template causing the issue

Now when you try to “Enable in Azure” it should work.

This also works with the other VMware types that can be enabled.

Updating Azure CLI on WSL2

I came across an issue on my local system when attempting to update Azure CLI to the latest version so that I could check out the Azure Arc HCIBox Jumpstart (It needs at least version 2.40.0).

I’m running Windows 11, with WSL2 Ubuntu-20.04 and I installed the AZ CLI using the one line install command. At the time, it was version 2.39.0

I tried to use the ‘az upgrade’ command, but it didn’t work.

So I went to the link provided https://aka.ms/doc/InstallAzureCli

There was nothing regarding the issue I had, so I figured I had to remove the cli and re-install.

Here’s the command I ran to remove the CLI

rm -r $HOME/lib/azure-cli
rm $HOME/bin/az
sed -i '/$HOME\/lib\/azure-cli\/az.completion/d' $HOME/.bash_profile
hash -r

I figured I should use apt for the re-install:

sudo apt-get update && sudo apt-get install --only-upgrade -y azure-cli

Checking the version shows I now have the latest version:

And it looks like running ‘az upgrade’ should work…

Cool! Now it’s time to check out the HCIBox ;)

Creating MAAS workload annotations for Kubernetes using kubectl output

Trying to manually keep track of servers and versions is challenging. MAAS has a workload annotation feature that allows you to create name=value using command line calls using MASS cli. I wanted to explore creating a script to pull data from kubectl and create workload annotations for machines in a Kubernetes cluster.

For this exercise, I intend to execute the script from a MAAS rack controller, on which I have installed kubectl and PowerShell. In the lab, there are three microk8s clusters. The first job was to collect the config files from each cluster and combine them into a single config so I can use contexts from within the kubectl commands. Using the command ‘kubectl config view --raw > kn.conf’ I created one file for each k8s cluster on the region controller in directory ~/.kube/clusters.

mkdir ~/.kube/clusters
mv /path/k1.conf ~/.kube/clusters
mv /path/k2.conf ~/.kube/clusters

export KUBECONFIG=$(find ~/.kube/clusters -type f | sed ':a;N;s/\n/:/;ba')

kubectl config get-clusters

kubectl config view --flatten > ~/.kube/config

There are various methods to combine and clean up k8s cluster contexts. I had to clean up the file to create unique cluster names as well as user token names. I manually edited the ~/.kube/config combined file and tweaked these settings.

This is the script I created. It does require that you have a MAAS profile created already.

# tag-maasworkloadannotations.ps1

[CmdletBinding()]
param (
    $k8scontext ,
    $maasProfile
)
write-verbose "executing tag kubernetes workload annotation for MAAS"
$LastUpdated = Get-Date

# Kubectl contexts
Write-Verbose "Retrieving Kubernetes cluster details from context $k8scontext"
$kcontexts = kubectl config view -o json | convertfrom-json
Write-Verbose "$(($kcontexts.contexts | Measure-Object).count) kube contexts found"
$kcontext = $kcontexts.contexts | Where-Object {$_.name -eq $k8scontext}
$kversion = kubectl version --context $k8scontext -o json | convertfrom-json

# K8s nodes
Write-Verbose "Retrieving Kubernetes nodes from context $k8scontext"
$nodes = kubectl get nodes --context $k8scontext -o json | convertfrom-json
Write-Verbose "$(($nodes.items | Measure-Object).count) k8s nodes found"

# Maas machines
Write-Verbose "Retrieving machines from MAAS using profile $maasProfile"
$machines = maas $maasProfile machines read | convertfrom-json
Write-Verbose "$($machines.count) maas machines found"
$powerParams  = maas $maasProfile machines power-parameters  | convertfrom-json

# Build Annotations
Write-Verbose "Building workload annotation records"
$WorkloadAnnontations = @()
foreach ($node in $nodes.items) {

    $WorkloadAnnontation = @{}
    $WARecord = '' | select-object systemid, hostname, WA 

    $machine = $machines | Where-Object {$_.hostname -eq $node.metadata.name} 
    if ($machine -ne ""){
        $WARecord.systemid = $machine.system_id
        $WARecord.hostname = $machine.hostname

        #$WorkloadAnnontation.add("osImage",  $node.status.nodeInfo.osImage)
        #$WorkloadAnnontation.add("systemUUID",  $node.status.nodeInfo.systemUUID)
        #$WorkloadAnnontation.add("machineID",  $node.status.nodeInfo.machineID)

        $WorkloadAnnontation.add("k8scluster", $kcontext.context.cluster )
        $WorkloadAnnontation.add("buildDate", $kversion.serverVersion.buildDate)
        $WorkloadAnnontation.add("containerRuntimeVersion",   $node.status.nodeInfo.containerRuntimeVersion)
        $WorkloadAnnontation.add("kernelVersion",   $node.status.nodeInfo.kernelVersion)
        $WorkloadAnnontation.add("kubeProxyVersion",   $node.status.nodeInfo.kubeProxyVersion)
        $WorkloadAnnontation.add("kubeletVersion",   $node.status.nodeInfo.kubeletVersion)
        $WorkloadAnnontation.add("outofband",  $powerParams.$($machine.system_id).power_address)
        $WorkloadAnnontation.add("AnnotationUpdated",$LastUpdated)

        if ($node.metadata.labels.'node.kubernetes.io/microk8s-controlplane' -eq "microk8s-controlplane")
            {$WorkloadAnnontation.add("nodeType", 'Master')}
        if ($node.metadata.labels.'node.kubernetes.io/microk8s-worker' -eq "microk8s-worker")
            {$WorkloadAnnontation.add("nodeType", 'Worker')}

        $WARecord.wa = $WorkloadAnnontation
    }
    $WorkloadAnnontations += $WARecord
} 

# publish workload annotations
$i = 1
$c = $($WorkloadAnnontations.count)
Write-Verbose "Publishing $c workload annotation records"
foreach ($WA in $WorkloadAnnontations){
    $KeyValueData = ""
    foreach ($key in $WA.wa.keys) {
        $KeyValueData += "$($key)='$($wa.wa.($key))' "
    }
    Write-Verbose "[$i/$c] Building command for $($WA.hostname)"
    $execmd = "maas $maasProfile machine set-workload-annotations $($WA.systemID) $KeyValueData > /dev/null 2>&1"
    write-debug $execmd
    Invoke-Expression $execmd
    Write-Verbose "[$i/$c] Command executed for $($WA.hostname)" 
    $i++ 
}

$RunTime = New-TimeSpan -Start $LastUpdated -End (get-date) 
$ExecutionTime = "Execution time was {0} hours, {1} minutes, {2} seconds and {3} milliseconds." -f $RunTime.Hours,  $RunTime.Minutes,  $RunTime.Seconds,  $RunTime.Milliseconds 
write-verbose $ExecutionTime

Using the [CmdletBinding()] allows me to leverage verbose and debug options.

./tag-maasworkloadannotations.ps1 -k8scontext k8sdemo -maasprofile mquick -Verbose -debug

View fullsize

./tag-maasworkloadannotations.ps1 -k8scontext microk8s -maasprofile mquick -Verbose

View fullsize

This took 12 minutes but could probably create streamlined script with PowerShell jobs

View fullsize

which can allow combinations of filters on the workload annotations

View fullsize

There is a lot more possible here, this was helpful for me while consolidating microk8s clusters and making sure I wasn’t releasing machines that were in use in a cluster if I had tagged them incorrectly in MAAS.

External Reference

Using multiple kubeconfig files and how to merge to a single – Oueta

Deploying ASDK 2206 to an Azure VM

Azure Stack Hub version 2206 was release a couple of months ago, but anyone trying to deploy ASDK 2206 to Azure will have found that the latest version is 2108. Until the official method is updated, here’s how you can do it .

I’m using the awesome scripts by Yagmur Sahin as the basis for the solution: https://github.com/yagmurs/AzureStack-VM-PoC

• Open the Azure portal and then create a PowerShell Cloud Shell.

View fullsize

I recommend you reset the user settings, as there can be issues with versions of the Azure PowerShell modules.

• Run the following command in the new PowerShell session:

git clone https://github.com/dmc-tech/AzureStack-VM-PoC.git

• Run the following, changing to meet your requirements. VirtualMachineSize can be from the following sizes:

  "Standard_E32s_v3",

  "Standard_E48s_v3"

cd ./AzureStack-VM-PoC/ARMv2

$ResourceGroupName = 'asdk01-uks'
$Region = 'uk south' 
$VirtualMachineSize = 'Standard_E48s_v3' 
$DataDiskCount = 11

./Deploy-AzureStackonAzureVM.ps1 -ResourceGroupName $ResourceGroupName -Region $Region  -VirtualMachineSize $VirtualMachineSize  -DataDiskCount $DataDiskCount

The configuration example above has enough resources to run an OpenShift cluster.

Running the script will initially:

• Create a resource group

• Create a storage account

• copy the ASDK 2206 VHD image to the storage account

• Create the VM using the VHD image

• Create a Public IP for the VM

Note: As part of the provisioning process, the admin user account you specify gets changed to ‘Administrator’. I would Strongly recommend removing the Public IP associated with the VM and deploy Azure Bastion to protect your ASDK instance

Once the ASDK VM has been provisioned, connect to it (Bastion or RDP). The username you specified previously is ignored, so use ‘Administrator’ as the user and enter the password you defined.

Once connected, open a PowerShell window (as Administrator), and run the following as an example (I’m using ADFS as I’m simulating a disconnected environment)

C:\CloudDeployment\Setup\InstallAzureStackPOC.ps1 -TimeServer '129.6.15.28' -DNSForwarder '8.8.8.8' -UseADFS

You’ll then need to enter the AdminPassword when prompted, and then the script will do it’s magic (as long as the password is correct!) and take a number of hours to install.

The above recording shows the first few minutes of the script (sped-up! :) ).

After a few hours, the VM will reboot. If you want to check progress, you should use the following username to connect:

azurestackadmin@azurestack.local

Use the password you initially defined

Here’s some of the output you’ll see from PowerShell if you do connect as azurestackadmin (there’s still a few hours left to go!)

After 7hours 25 minutes, the install completed. You can determine this from the following log entry:

To prove that version 2206 has been installed, open the admin portal and check the properties for the region/instance.

As I used ADFS for this example, I had to login as cloudadmin@azurestack.local. If using AAD use the account you define when initially running the setup script.

View fullsize

Hope that helps if you want to deploy version 2206 as well as a simplified deployment tutorial.

Footnote: I tried using v5 series VM’s to deploy ASDK on, but it failed due to a network issue. I assume it is due to a different NIC/drive being used than the v3 series.

Azure Arc Connected K8s Billing usage workbook error - 'union' operator: Failed to resolve table expression named 'ContainerLogV2'

Using ‘Container Insights’ for Kubernetes can get expensive, I recently saw a customer creating 19TB of logs a month from logging container insights from an AKS cluster.

You may want to check what logs you are capturing as suggested through Microsoft guides Monitoring cost for Container insights - Azure Monitor | Microsoft Docs which suggests one of the first steps to check billing usage.

As of writing this if you enable container insights and try to view Data Usage you get an error. ‘union' operator: Failed to resolve table expression named 'ContainerLogV2’. ContainerLogV2 is in preview or if you are just aren’t ready to deploy that, here is how to fix the report manually. I also have included the Full report at the end.

View fullsize

However, depending on when you deploy this and how you deployed insight metrics you may hit this error.

As of writing this, ContainerLogV2 is in preview and you can follow this guide to deploy it. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs. If you want to gain some quick insight into your usage you can edit the workbook. We just need to get through the maze of edits to get to the underlying query.

• Select ‘Azure Arc’ services

• select ‘Kubernetes clusters’ under infrastructure

• select your K8s cluster

• select ‘Workbooks’ under Monitoring

• select ‘Container Insights Usage’ workbook

• you can either edit this one or ‘Save As’ and create a new name. I will create a copy.

View fullsize

Open your chosen workbook

you can now edit the group item.

by Namespace

Editing the Namespace selection Dropdown parameter query

Hopefully, you get the gist of the process. There are a few more queries to edit.

And

This will now work for Data Usage. This is likely a temporary solution until ContainerLogV2 comes out of preview. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs

Gallery Template code

{
  "version": "Notebook/1.0",
  "items": [
    {
      "type": 9,
      "content": {
        "version": "KqlParameterItem/1.0",
        "crossComponentResources": [
          "{resource}"
        ],
        "parameters": [
          {
            "id": "670aac26-0ffe-4f29-81c2-d48911bc64b6",
            "version": "KqlParameterItem/1.0",
            "name": "timeRange",
            "label": "Time Range",
            "type": 4,
            "description": "Select time-range for data selection",
            "isRequired": true,
            "value": {
              "durationMs": 21600000
            },
            "typeSettings": {
              "selectableValues": [
                {
                  "durationMs": 300000
                },
                {
                  "durationMs": 900000
                },
                {
                  "durationMs": 3600000
                },
                {
                  "durationMs": 14400000
                },
                {
                  "durationMs": 43200000
                },
                {
                  "durationMs": 86400000
                },
                {
                  "durationMs": 172800000
                },
                {
                  "durationMs": 259200000
                },
                {
                  "durationMs": 604800000
                },
                {
                  "durationMs": 1209600000
                },
                {
                  "durationMs": 2419200000
                },
                {
                  "durationMs": 2592000000
                },
                {
                  "durationMs": 5184000000
                },
                {
                  "durationMs": 7776000000
                }
              ],
              "allowCustom": true
            }
          },
          {
            "id": "bfc96857-81df-4f0d-b958-81f96d28ddeb",
            "version": "KqlParameterItem/1.0",
            "name": "resource",
            "type": 5,
            "isRequired": true,
            "isHiddenWhenLocked": true,
            "typeSettings": {
              "additionalResourceOptions": [
                "value::1"
              ],
              "showDefault": false
            },
            "timeContext": {
              "durationMs": 21600000
            },
            "timeContextFromParameter": "timeRange",
            "defaultValue": "value::1"
          },
          {
            "id": "8d48ec94-fde6-487c-98bf-f1295f5d8b81",
            "version": "KqlParameterItem/1.0",
            "name": "resourceType",
            "type": 7,
            "description": "Resource type of resource",
            "isRequired": true,
            "query": "{\"version\":\"1.0.0\",\"content\":\"\\\"{resource:resourcetype}\\\"\",\"transformers\":null}",
            "isHiddenWhenLocked": true,
            "typeSettings": {
              "additionalResourceOptions": [
                "value::1"
              ],
              "showDefault": false
            },
            "timeContext": {
              "durationMs": 21600000
            },
            "timeContextFromParameter": "timeRange",
            "defaultValue": "value::1",
            "queryType": 8
          },
          {
            "id": "1b826776-ab99-45ab-86db-cced05e8b36d",
            "version": "KqlParameterItem/1.0",
            "name": "clusterId",
            "type": 1,
            "description": "Used to identify the cluster name when the cluster type is AKS Engine",
            "isHiddenWhenLocked": true,
            "timeContext": {
              "durationMs": 0
            },
            "timeContextFromParameter": "timeRange"
          },
          {
            "id": "67227c35-eab8-4518-9212-1c3c3d564b20",
            "version": "KqlParameterItem/1.0",
            "name": "masterNodeExists",
            "type": 1,
            "query": "let MissingTable = view () { print isMissing=1 };\r\nlet masterNodeExists = toscalar(\r\nunion isfuzzy=true MissingTable, (\r\nAzureDiagnostics \r\n| getschema \r\n| summarize c=count() \r\n| project isMissing=iff(c > 0, 0, 1)\r\n) \r\n| top 1 by isMissing asc\r\n);\r\nprint(iif(masterNodeExists == 0, 'yes', 'no'))\r\n",
            "crossComponentResources": [
              "{resource}"
            ],
            "isHiddenWhenLocked": true,
            "timeContext": {
              "durationMs": 0
            },
            "timeContextFromParameter": "timeRange",
            "queryType": 0,
            "resourceType": "{resourceType}"
          }
        ],
        "style": "pills",
        "queryType": 0,
        "resourceType": "{resourceType}"
      },
      "name": "pills"
    },
    {
      "type": 1,
      "content": {
        "json": "Please note that the Container Insights Usage workbook for AKS Engine clusters shows your billable data usage for your cluster's entire workspace ({resource:name}), and not just the cluster itself ({clusterId}). ",
        "style": "info"
      },
      "conditionalVisibility": {
        "parameterName": "resourceType",
        "comparison": "isEqualTo",
        "value": "microsoft.operationalinsights/workspaces"
      },
      "name": "aks-engine-billable-data-shown-applies-to-entire-workspace-not-just-the-cluster-info-message",
      "styleSettings": {
        "showBorder": true
      }
    },
    {
      "type": 11,
      "content": {
        "version": "LinkItem/1.0",
        "style": "tabs",
        "links": [
          {
            "id": "3b7d39f2-38c5-4586-a155-71e28e333020",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "Overview",
            "subTarget": "overview",
            "style": "link"
          },
          {
            "id": "7163b764-7ab2-48b3-b417-41af3eea7ef0",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Table",
            "subTarget": "table",
            "style": "link"
          },
          {
            "id": "960bbeea-357f-4c07-bdd9-6f3f123d56fd",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Namespace",
            "subTarget": "namespace",
            "style": "link"
          },
          {
            "id": "7fbcc5bd-62d6-4aeb-b55e-7b1fab65716a",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Log Source",
            "subTarget": "logSource",
            "style": "link"
          },
          {
            "id": "3434a63b-f3d5-4c11-9620-1e731983041c",
            "cellValue": "selectedTab",
            "linkTarget": "parameter",
            "linkLabel": "By Diagnostic Master Node ",
            "subTarget": "diagnosticMasterNode",
            "style": "link"
          }
        ]
      },
      "name": "tabs"
    },
    {
      "type": 1,
      "content": {
        "json": "Due to querying limitation on Basic Logs, if you are using Basic Logs, this workbook provides partial data on your container log usage.",
        "style": "info"
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "overview"
      },
      "name": "basic-logs-info-text"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "title": "Container Insights Billing Usage",
        "items": [
          {
            "type": 1,
            "content": {
              "json": "<br/>\r\nThis workbook provides you a summary and source for billable data collected by [Container Insights solution.](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview)\r\n\r\nThe best way to understand how Container Insights ingests data in Log Analytics workspace is from this [article.](https://medium.com/microsoftazure/azure-monitor-for-containers-optimizing-data-collection-settings-for-cost-ce6f848aca32)\r\n\r\nIn this workbook you can:\r\n\r\n* View billable data ingested by **solution**.\r\n* View billable data ingested by **Container logs (application logs)**\r\n* View billable container logs data ingested segregated by **Kubernetes namespace**\r\n* View billable container logs data ingested segregated by **Cluster name**\r\n* View billable container log data ingested by **logsource entry**\r\n* View billable diagnostic data ingested by **diagnostic master node logs**\r\n\r\nYou can fine tune and control logging by turning off logging on the above mentioned vectors. [Learn how to fine-tune logging](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config)\r\n\r\nYou can control your master node's logs by updating diagnostic settings. [Learn how to update diagnostic settings](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "name": "workbooks-explanation-text"
          },
          {
            "type": 1,
            "content": {
              "json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "no"
            },
            "name": "master-node-logs-are-not-enabled-msg"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "overview"
      },
      "name": "workbook-explanation"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "union withsource = SourceTable AzureDiagnostics, AzureActivity, AzureMetrics, ContainerLog, Perf, KubePodInventory, ContainerInventory, InsightsMetrics, KubeEvents, KubeServices, KubeNodeInventory, ContainerNodeInventory, KubeMonAgentEvents, ContainerServiceLog, Heartbeat, KubeHealth, ContainerImageInventory\r\n| where _IsBillable == true\r\n| project _BilledSize, TimeGenerated, SourceTable\r\n| summarize BillableDataBytes = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), SourceTable\r\n| render piechart\r\n\r\n\r\n",
              "size": 3,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Data from Container Insights",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "xAxis": "TimeGenerated",
                "createOtherGroup": 100,
                "seriesLabelSettings": [
                  {
                    "seriesName": "LogManagement",
                    "label": "LogManagementSolution(GB)"
                  },
                  {
                    "seriesName": "ContainerInsights",
                    "label": "ContainerInsightsSolution(GB)"
                  }
                ],
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "50",
            "showPin": true,
            "name": "billable-data-from-ci"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "table"
      },
      "name": "by-datatable-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| summarize Total=sum(BillableDataBytes) by Namespace\r\n| render piechart\r\n",
              "size": 3,
              "showAnalytics": true,
              "title": "Billable Container Log Data Per Namespace",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "tileSettings": {
                "showBorder": false,
                "titleContent": {
                  "columnMatch": "Namespace",
                  "formatter": 1
                },
                "leftContent": {
                  "columnMatch": "Total",
                  "formatter": 12,
                  "formatOptions": {
                    "palette": "auto"
                  },
                  "numberFormat": {
                    "unit": 17,
                    "options": {
                      "maximumSignificantDigits": 3,
                      "maximumFractionDigits": 2
                    }
                  }
                }
              },
              "graphSettings": {
                "type": 0,
                "topContent": {
                  "columnMatch": "Namespace",
                  "formatter": 1
                },
                "centerContent": {
                  "columnMatch": "Total",
                  "formatter": 1,
                  "numberFormat": {
                    "unit": 17,
                    "options": {
                      "maximumSignificantDigits": 3,
                      "maximumFractionDigits": 2
                    }
                  }
                }
              },
              "chartSettings": {
                "createOtherGroup": 100,
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "name": "billable-data-per-namespace"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "namespace"
      },
      "name": "by-namespace-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 9,
            "content": {
              "version": "KqlParameterItem/1.0",
              "crossComponentResources": [
                "{resource}"
              ],
              "parameters": [
                {
                  "id": "d5701caa-0486-4e6f-adad-6fa5b8496a7d",
                  "version": "KqlParameterItem/1.0",
                  "name": "namespace",
                  "label": "Namespace",
                  "type": 2,
                  "description": "Filter data by namespace",
                  "isRequired": true,
                  "query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| project ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| distinct Namespace\r\n| project value = Namespace, label = Namespace, selected = false\r\n| sort by label asc",
                  "crossComponentResources": [
                    "{resource}"
                  ],
                  "typeSettings": {
                    "additionalResourceOptions": [
                      "value::1"
                    ],
                    "showDefault": false
                  },
                  "timeContext": {
                    "durationMs": 0
                  },
                  "timeContextFromParameter": "timeRange",
                  "defaultValue": "value::1",
                  "queryType": 0,
                  "resourceType": "{resourceType}"
                }
              ],
              "style": "pills",
              "queryType": 0,
              "resourceType": "{resourceType}"
            },
            "name": "namespace-pill"
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| join hint.strategy=shuffle (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by LogEntrySource, ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n)\r\n| extend sourceNamespace = strcat(LogEntrySource, \"/\", Namespace)\r\n| summarize Total=sum(BillableDataBytes) by sourceNamespace\r\n| render piechart",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Data Per Log-Source Type for Namespace Selected",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "createOtherGroup": 100,
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "35",
            "showPin": true,
            "name": "billable-data-per-log-source",
            "styleSettings": {
              "showBorder": true
            }
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n| join hint.strategy=shuffle ( \r\nContainerLog\r\n| project _BilledSize, ContainerID, TimeGenerated, LogEntrySource\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n)\r\n| summarize ContainerLogData = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), LogEntrySource\r\n| render linechart\r\n\r\n",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Container Log Data for Namespace Selected",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "xSettings": {},
                "ySettings": {
                  "numberFormatSettings": {
                    "unit": 2,
                    "options": {
                      "style": "decimal"
                    }
                  }
                }
              }
            },
            "customWidth": "65",
            "conditionalVisibility": {
              "parameterName": "namespace:label",
              "comparison": "isNotEqualTo",
              "value": "<unset>"
            },
            "showPin": true,
            "name": "billable-container-log-data-for-namespace-selected",
            "styleSettings": {
              "showBorder": true
            }
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "logSource"
      },
      "name": "by-log-source-tab"
    },
    {
      "type": 12,
      "content": {
        "version": "NotebookGroup/1.0",
        "groupType": "editable",
        "items": [
          {
            "type": 1,
            "content": {
              "json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
            },
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "no"
            },
            "name": "master-node-logs-are-not-enabled-msg-under-masternode-tab"
          },
          {
            "type": 3,
            "content": {
              "version": "KqlItem/1.0",
              "query": "AzureDiagnostics\r\n| summarize Total=sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), Category\r\n| render barchart\r\n\r\n\r\n",
              "size": 0,
              "showAnnotations": true,
              "showAnalytics": true,
              "title": "Billable Diagnostic Master Node Log Data",
              "timeContextFromParameter": "timeRange",
              "queryType": 0,
              "resourceType": "{resourceType}",
              "crossComponentResources": [
                "{resource}"
              ],
              "chartSettings": {
                "ySettings": {
                  "unit": 2,
                  "min": null,
                  "max": null
                }
              }
            },
            "customWidth": "75",
            "conditionalVisibility": {
              "parameterName": "masterNodeExists",
              "comparison": "isEqualTo",
              "value": "yes"
            },
            "showPin": true,
            "name": "billable-data-from-diagnostic-master-node-logs"
          }
        ]
      },
      "conditionalVisibility": {
        "parameterName": "selectedTab",
        "comparison": "isEqualTo",
        "value": "diagnosticMasterNode"
      },
      "name": "by-diagnostic-master-node-tab"
    }
  ],
  "fallbackResourceIds": [
    "/subscriptions/357e7522-d1ad-433c-9545-7d8b8d72d61a/resourceGroups/ArcResources/providers/Microsoft.Kubernetes/connectedClusters/sv5-k8sdemo"
  ],
  "fromTemplateId": "community-Workbooks/AKS/Billing Usage Divided",
  "$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json"
}