Trying to manually keep track of servers and versions is challenging. MAAS has a workload annotation feature that allows you to create name=value using command line calls using MASS cli. I wanted to explore creating a script to pull data from kubectl and create workload annotations for machines in a Kubernetes cluster.
For this exercise, I intend to execute the script from a MAAS rack controller, on which I have installed kubectl and PowerShell. In the lab, there are three microk8s clusters. The first job was to collect the config files from each cluster and combine them into a single config so I can use contexts from within the kubectl commands. Using the command ‘kubectl config view --raw > kn.conf’ I created one file for each k8s cluster on the region controller in directory ~/.kube/clusters.
mkdir ~/.kube/clusters
mv /path/k1.conf ~/.kube/clusters
mv /path/k2.conf ~/.kube/clusters
export KUBECONFIG=$(find ~/.kube/clusters -type f | sed ':a;N;s/\n/:/;ba')
kubectl config get-clusters
kubectl config view --flatten > ~/.kube/configThere are various methods to combine and clean up k8s cluster contexts. I had to clean up the file to create unique cluster names as well as user token names. I manually edited the ~/.kube/config combined file and tweaked these settings.
This is the script I created. It does require that you have a MAAS profile created already.
# tag-maasworkloadannotations.ps1
[CmdletBinding()]
param (
$k8scontext ,
$maasProfile
)
write-verbose "executing tag kubernetes workload annotation for MAAS"
$LastUpdated = Get-Date
# Kubectl contexts
Write-Verbose "Retrieving Kubernetes cluster details from context $k8scontext"
$kcontexts = kubectl config view -o json | convertfrom-json
Write-Verbose "$(($kcontexts.contexts | Measure-Object).count) kube contexts found"
$kcontext = $kcontexts.contexts | Where-Object {$_.name -eq $k8scontext}
$kversion = kubectl version --context $k8scontext -o json | convertfrom-json
# K8s nodes
Write-Verbose "Retrieving Kubernetes nodes from context $k8scontext"
$nodes = kubectl get nodes --context $k8scontext -o json | convertfrom-json
Write-Verbose "$(($nodes.items | Measure-Object).count) k8s nodes found"
# Maas machines
Write-Verbose "Retrieving machines from MAAS using profile $maasProfile"
$machines = maas $maasProfile machines read | convertfrom-json
Write-Verbose "$($machines.count) maas machines found"
$powerParams = maas $maasProfile machines power-parameters | convertfrom-json
# Build Annotations
Write-Verbose "Building workload annotation records"
$WorkloadAnnontations = @()
foreach ($node in $nodes.items) {
$WorkloadAnnontation = @{}
$WARecord = '' | select-object systemid, hostname, WA
$machine = $machines | Where-Object {$_.hostname -eq $node.metadata.name}
if ($machine -ne ""){
$WARecord.systemid = $machine.system_id
$WARecord.hostname = $machine.hostname
#$WorkloadAnnontation.add("osImage", $node.status.nodeInfo.osImage)
#$WorkloadAnnontation.add("systemUUID", $node.status.nodeInfo.systemUUID)
#$WorkloadAnnontation.add("machineID", $node.status.nodeInfo.machineID)
$WorkloadAnnontation.add("k8scluster", $kcontext.context.cluster )
$WorkloadAnnontation.add("buildDate", $kversion.serverVersion.buildDate)
$WorkloadAnnontation.add("containerRuntimeVersion", $node.status.nodeInfo.containerRuntimeVersion)
$WorkloadAnnontation.add("kernelVersion", $node.status.nodeInfo.kernelVersion)
$WorkloadAnnontation.add("kubeProxyVersion", $node.status.nodeInfo.kubeProxyVersion)
$WorkloadAnnontation.add("kubeletVersion", $node.status.nodeInfo.kubeletVersion)
$WorkloadAnnontation.add("outofband", $powerParams.$($machine.system_id).power_address)
$WorkloadAnnontation.add("AnnotationUpdated",$LastUpdated)
if ($node.metadata.labels.'node.kubernetes.io/microk8s-controlplane' -eq "microk8s-controlplane")
{$WorkloadAnnontation.add("nodeType", 'Master')}
if ($node.metadata.labels.'node.kubernetes.io/microk8s-worker' -eq "microk8s-worker")
{$WorkloadAnnontation.add("nodeType", 'Worker')}
$WARecord.wa = $WorkloadAnnontation
}
$WorkloadAnnontations += $WARecord
}
# publish workload annotations
$i = 1
$c = $($WorkloadAnnontations.count)
Write-Verbose "Publishing $c workload annotation records"
foreach ($WA in $WorkloadAnnontations){
$KeyValueData = ""
foreach ($key in $WA.wa.keys) {
$KeyValueData += "$($key)='$($wa.wa.($key))' "
}
Write-Verbose "[$i/$c] Building command for $($WA.hostname)"
$execmd = "maas $maasProfile machine set-workload-annotations $($WA.systemID) $KeyValueData > /dev/null 2>&1"
write-debug $execmd
Invoke-Expression $execmd
Write-Verbose "[$i/$c] Command executed for $($WA.hostname)"
$i++
}
$RunTime = New-TimeSpan -Start $LastUpdated -End (get-date)
$ExecutionTime = "Execution time was {0} hours, {1} minutes, {2} seconds and {3} milliseconds." -f $RunTime.Hours, $RunTime.Minutes, $RunTime.Seconds, $RunTime.Milliseconds
write-verbose $ExecutionTimeUsing the [CmdletBinding()] allows me to leverage verbose and debug options.
./tag-maasworkloadannotations.ps1 -k8scontext k8sdemo -maasprofile mquick -Verbose -debug./tag-maasworkloadannotations.ps1 -k8scontext microk8s -maasprofile mquick -Verbose
This took 12 minutes but could probably create streamlined script with PowerShell jobs
which can allow combinations of filters on the workload annotations
There is a lot more possible here, this was helpful for me while consolidating microk8s clusters and making sure I wasn’t releasing machines that were in use in a cluster if I had tagged them incorrectly in MAAS.
External Reference
Using multiple kubeconfig files and how to merge to a single – Oueta
