Crying Cloud

View Original

How to calculate Azure VHDs used space

Giovanni Fleres

One of the most hotly topic in the Azure world, is estimate how much storage is currently used by deployed VMs. I have intentionally used the word "used" and not allocated because in Azure, when VHDs are stored a Standard Storage Account, they're like "thin" or dynamically disk if you prefer, you aren't really using all the allocated space and Azure portal confirms this. Below image shows how an Azure VHD of 127 GB used as OS disk is viewed from a Windows VM

Below image shows how Azure portal calculate usage space for above disk

For reporting and billing reason, you may need to get these information for all VMs deployed in a specific subscription.

This article will show how to retrieve these information for VHDs stored in both Standard Account and Premium Account.

A little of Azure theory..

Standard Storage Account:

When a new Azure Storage Account is created, by default, some hidden tables are created and one of these is the "$MetricsCapacityBlob". This table shows blobs capacity values.

Note: There others hidden tables which contains other info related to an Azure Storage Account like its transactions.

Premium Storage Account:

From Microsoft Web Site: "Billing for a premium storage disk/blob depends on the provisioned size of the disk/blob. Azure maps the provisioned size (rounded up) to the nearest premium storage disk option as specified in the table given in the Scalability and Performance Targets when using Premium Storage section. Each disk will map to one of the the supported provisioned sizes and will be billed accordingly. Billing for any provisioned disk is prorated hourly using the monthly price for the Premium Storage offer. For example, if you provisioned a P10 disk and deleted it after 20 hours, you are billed for the P10 offering prorated to 20 hours. This is regardless of the amount of actual data written to the disk or the IOPS/throughput used."

From a reporting point of view, this mean that the size of a deployed VHD matches the allocated space and you're billed for its size "regardless of the amount of actual data written to the disk or the IOPS/throughput used".

Before to start to write some PowerShell code, it's required to prepare your workstation to run the Azure Storage Report:

  • If OS is older then Windows Server 2016 or Windows 10, then it's required to download and install PowerShell 5.0 from here
  • Install ReportHTML module from PowerShell Gallery: Open a PowerShell console as administrator and execute the following code

[powershell]

Install-Module -Name ReportHTML

[/powershell]

Let's begin to write some PowerShell code

Note: Most of the below functions come from Get Billable Size of Windows Azure Blobs (w/Snapshots) in a Container or Account script developed by the Windows Azure Product Team Scripts. Their code has been updated to work with latest Azure PowerShell module and support script purpose

Open a PowerShell editor and create a new file called Module-Azure.ps1

This file will contain all functions invoked by the main script

[powershell] function global:Connect-Azure {

Login-AzureRmAccount

$subName = Get-AzureRmSubscription | select SubscriptionName | Out-GridView -Title "Select a subscription" -OutputMode Single | select -ExpandProperty SubscriptionName

Select-AzureRmSubscription -SubscriptionName $subName

$global:azureSubscription = Get-AzurermSubscription -SubscriptionName $subName

}

function global:Calculate-BlobSpace {

param( # The name of the storage account to enumerate. [Parameter(Mandatory = $true)] [string]$StorageAccountName ,

# The name of the storage container to enumerate. [Parameter(Mandatory = $false)] [ValidateNotNullOrEmpty()] [string]$ContainerName,

# The name of the storage account resource group. [Parameter(Mandatory = $true)] [ValidateNotNullOrEmpty()] [string] $StorageAccountRGName )

# Following modifies the Write-Verbose behavior to turn the messages on globally for this session $VerbosePreference = "Continue"

$storageAccount = Get-AzureRmStorageAccount -ResourceGroupName $StorageAccountRGName -Name $StorageAccountName -ErrorAction SilentlyContinue

if ($storageAccount -eq $null) { throw "The storage account specified does not exist in this subscription." }

# Instantiate a storage context for the storage account. $storagePrimaryKey = ((Get-AzureRmStorageAccountKey -ResourceGroupName $StorageAccountRGName -Name $StorageAccountName)[0]).Value

$storageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $storagePrimaryKey

# Get a list of containers to process. $containers = New-Object System.Collections.ArrayList if ($ContainerName.Length -ne 0) { $container = Get-AzureStorageContainer -Context $storageContext ` -Name $ContainerName -ErrorAction SilentlyContinue | ForEach-Object { $containers.Add($_) } | Out-Null } else { Get-AzureStorageContainer -Context $storageContext -ErrorAction SilentlyContinue | ForEach-Object { $containers.Add($_) } | Out-Null }

# Calculate size. $sizeInBytes = 0 if ($containers.Count -gt 0) { $containers | ForEach-Object { $result = Get-ContainerBytes $_.CloudBlobContainer $sizeInBytes += $result.containerSize Write-Verbose ("Container '{0}' with {1} blobs has a size of {2:F2}MB." -f ` $_.CloudBlobContainer.Name, $result.blobCount, ($result.containerSize / 1MB)) } foreach ($container in $containers) {

$result = Get-ContainerBytes $container.CloudBlobContainer

$sizeInBytes += $result.containerSize

Write-Verbose ("Container '{0}' with {1} blobs has a size of {2:F2}MB." -f $container.CloudBlobContainer.Name, $result.blobCount, ($result.containerSize / 1MB)) }

$sizeInGB = [math]::Round($sizeInBytes / 1GB)

return $sizeInGB } else { Write-Warning "No containers found to process in storage account '$StorageAccountName'."

$sizeInGB = 0

return $sizeInGB } }

function global:Get-BlobBytes { param ( [Parameter(Mandatory=$true)] [Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageBlob]$Blob)

# Base + blob name $blobSizeInBytes = 124 + $Blob.Name.Length * 2

# Get size of metadata $metadataEnumerator = $Blob.ICloudBlob.Metadata.GetEnumerator() while ($metadataEnumerator.MoveNext()) { $blobSizeInBytes += 3 + $metadataEnumerator.Current.Key.Length + $metadataEnumerator.Current.Value.Length }

if ($Blob.BlobType -eq [Microsoft.WindowsAzure.Storage.Blob.BlobType]::BlockBlob) { $blobSizeInBytes += 8 $Blob.ICloudBlob.DownloadBlockList() | ForEach-Object { $blobSizeInBytes += $_.Length + $_.Name.Length } } else { $Blob.ICloudBlob.GetPageRanges() | ForEach-Object { $blobSizeInBytes += 12 + $_.EndOffset - $_.StartOffset } }

return $blobSizeInBytes }

function global:Get-ContainerBytes { param ( [Parameter(Mandatory=$true)] [Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer]$Container)

# Base + name of container $containerSizeInBytes = 48 + $Container.Name.Length * 2

# Get size of metadata $metadataEnumerator = $Container.Metadata.GetEnumerator() while ($metadataEnumerator.MoveNext()) { $containerSizeInBytes += 3 + $metadataEnumerator.Current.Key.Length + $metadataEnumerator.Current.Value.Length }

# Get size for Shared Access Policies $containerSizeInBytes += $Container.Permission.SharedAccessPolicies.Count * 512

# Calculate size of all blobs. $blobCount = 0 $blobs = Get-AzureStorageBlob -Context $storageContext -Container $Container.Name foreach ($blobItem in $blobs) { #$blobItem | Get-Member

$containerSizeInBytes += Get-BlobBytes $blobItem

$blobCount++

}

return @{ "containerSize" = $containerSizeInBytes; "blobCount" = $blobCount } }

function global:ListBlobCapacity([System.Array]$arr, $StgAccountName, $stgAccountRGName) {

$Delimiter = ','

$Today = Get-Date

$storageAccountKey = ((Get-AzureRmStorageAccountKey -ResourceGroupName $stgAccountRGName -Name $StgAccountName)[0]).Value

$StorageCtx = New-AzureStorageContext –StorageAccountName $StgAccountName –StorageAccountKey $StorageAccountKey

$metrics = Get-AzureStorageServiceMetricsProperty -Context $StorageCtx -ServiceType "Blob" -MetricsType Hour -ErrorAction "SilentlyContinue"

# if storage account has Monitoring turned on, get the Capacity for the configured nbr of Retention Days if ( $metrics.MetricsLevel -ne "None" ) { $RetentionDays = $metrics.RetentionDays if ( $RetentionDays -eq $null -or $RetentionDays -eq '' ) { $RetentionDays = 0 } $table = GetTableReference $StgAccountName $StorageAccountKey '$MetricsCapacityBlob' # loop over days for( $d = $RetentionDays; $d -ge 0; $d = $d - 1) {

$date = (Get-Date $Today.AddDays(-$d) -format 'yyyyMMdd')

$partitionKey = $date + "T0000"

$result = $table.Execute([Microsoft.WindowsAzure.Storage.Table.TableOperation]::Retrieve($partitionKey, "data")) if ( $result.HttpStatusCode -eq "200") { $arr += CreateRowObject $StgAccountName (Get-Date $Today.AddDays(-$d)).ToString("d") } } } return $arr }

function global:GetBlobsCurrentCapacity($StgAccountName, $stgAccountRGName) {

$Delimiter = ','

$Today = Get-Date

$storageAccountKey = ((Get-AzureRmStorageAccountKey -ResourceGroupName $stgAccountRGName -Name $StgAccountName)[0]).Value

$StorageCtx = New-AzureStorageContext –StorageAccountName $StgAccountName –StorageAccountKey $StorageAccountKey

$metrics = Get-AzureStorageServiceMetricsProperty -Context $StorageCtx -ServiceType "Blob" -MetricsType Hour -ErrorAction "SilentlyContinue"

# if storage account has Monitoring turned on, get the Capacity for the configured nbr of Retention Days if ( $metrics.MetricsLevel -ne "None" ) { $table = GetTableReference $StgAccountName $StorageAccountKey '$MetricsCapacityBlob'

$date = (Get-Date $Today.AddDays(-1) -format 'yyyyMMdd')

$partitionKey = $date + "T0000"

$result = $table.Execute([Microsoft.WindowsAzure.Storage.Table.TableOperation]::Retrieve($partitionKey, "data"))

if ( $result.HttpStatusCode -eq "200") { $rowObj = CreateRowObject $StgAccountName (Get-Date $Today.AddDays(-1)).ToString("d") }

}

return $rowObj }

# setup access to Azure Table $TableName function global:GetTableReference($StgAccountName, $StorageAccountKey, $TableName) { $accountCredentials = New-Object "Microsoft.WindowsAzure.Storage.Auth.StorageCredentials" $StgAccountName, $StorageAccountKey $storageAccount = New-Object "Microsoft.WindowsAzure.Storage.CloudStorageAccount" $accountCredentials, $true $tableClient = $storageAccount.CreateCloudTableClient() $table = $tableClient.GetTableReference($TableName) return $table }

function global:CreateRowObject($StgAccountName, $DateTime) { $row = New-Object System.Object $row | Add-Member -type NoteProperty -name "StorageAccountName" -Value $StgAccountName $row | Add-Member -type NoteProperty -name "DateTime" -Value $DateTime foreach( $key in $result.Result.Properties.Keys ) { $val = $result.Result.Properties[$key].PropertyAsObject

if ( $Delimiter -eq ",") { $val = $val -replace ",","." } $row | Add-Member -type NoteProperty -name $key -Value $val } return $row }

function global:get-PremiumBlobGBSize { param ( [Microsoft.WindowsAzure.Commands.Common.Storage.ResourceModel.AzureStorageBlob] $blobobj )

$blobGBSize = [math]::Truncate(($blobObj.Length / 1GB))

return $blobGBSize

}

[/powershell]

Some comments:

  • All functions have been declared as global to be invoked from main script if required
  • Connect-Azure: Allow to select the Azure subscription against which execute reporting script and establish a connection
  • Calculate-BlobSpace: This is the function invoked by the main script which returns the sum of the spaces allocated to VHDs for a given Standard Storage Account
  • Get-PremiumBlobGBSize:  This is the function invoked by the main script which returns the sum of the spaces allocated to VHDs for a given Premium Storage Account

Now save Module-Azure.ps1 and in the same directory where it has been saved, create a new PowerShell file called "Generate-AzureReport.ps1". This will be the main file which will invoke Module-Azure functions.

Open Generate-AzureReport.ps1 with a PowerShell editor and paste the following code:

 

[powershell]

$ScriptDir = $PSScriptRoot

Write-Host "Current script directory is $ScriptDir"

Set-Location -Path $ScriptDir

.\module-azure.ps1

Connect-Azure

if (!(get-module ReportHTML)) { if (!( get-module -ListAvailable)) { write-host "Please Install ReportHTML module from PowerShell Gallery" } else { Write-Host "Importing Report-HTML module"

Import-Module ReportHTML } } else { Write-Host "Report-HTML module is already installed" }

$subname = $azureSubscription.SubscriptionName

$billingReportFolder = "C:\temp\billing"

if ( !(test-path $billingReportFolder) ) { New-Item $billingReportFolder -ItemType Directory }

# Analyzing Standard Storage Account Consumptions from Azure Storage Hiden Table $MetricsCapacityBlob

$sa = Find-AzureRmResource -ResourceType Microsoft.Storage/storageAccounts | Where-Object {$_.Sku.tier -ne "Premium" }

$saConsumptions = @()

foreach ($saItem in $sa) { $blobObj = GetBlobsCurrentCapacity -StgAccountName $saItem.Name -stgAccountRGName $saItem.ResourceGroupName

$blobCapacityGB = [math]::Truncate(($blobObj.Capacity / 1GB))

$blobSpaceItem = '' | select StorageAccountName,Allocated_GB

$blobSpaceItem.StorageAccountName = $saItem.Name

$blobSpaceItem.Allocated_GB = $blobCapacityGB

$saConsumptions += $blobSpaceItem

}

$saPremium = Find-AzureRmResource -ResourceType Microsoft.Storage/storageAccounts | Where-Object {$_.Sku.tier -eq "Premium" }

$saPremiumUsage = @()

foreach ($saPremiumItem in $saPremium) { $storageAccountKey = ((Get-AzureRmStorageAccountKey -ResourceGroupName $saPremiumItem.ResourceGroupName -Name $saPremiumItem.Name)[0]).Value

$StorageCtx = New-AzureStorageContext –StorageAccountName $saPremiumItem.Name –StorageAccountKey $StorageAccountKey

$containers = Get-AzureStorageContainer -Context $StorageCtx

$saPremiumUsageItem = '' | select StorageAccountName,Allocated_GB

$saPremiumUsageItem.StorageAccountName = $saPremiumItem.Name

$saPremiumUsageItem.Allocated_GB = 0

foreach ($container in $containers) { $blobs = Get-AzureStorageBlob -Context $StorageCtx -Container $container.Name

foreach ($blobItem in $blobs) { $blobsize = get-PremiumBlobGBSize ($blobItem)

$saPremiumUsageItem.Allocated_GB = $saPremiumUsageItem.Allocated_GB + $blobsize } }

$saPremiumUsage += $saPremiumUsageItem

}

# Calculate Totals

$saConsumptionsTotal = 0

foreach ($saConsumptionsItem in $saConsumptions) { $saConsumptionsTotal = $saConsumptionsTotal + $saConsumptionsItem.Allocated_GB }

$saPremiumUsageTotal = 0

foreach ($saPremiumUsageItem in $saPremiumUsage) { $saPremiumUsageTotal = $saPremiumUsageTotal + $saPremiumUsageItem.Allocated_GB }

# Generate Reports

$Rpt = @()

$TitleText = "Azure Usage Report "

$Rpt += Get-HTMLOpenPage -TitleText $TitleText -LeftLogoName "sample"

##

$Rpt += Get-HtmlContentOpen -HeaderText "Standard Storage Accounts Consumptions (GBs)"

$saConsumptionsTableStyle = Set-TableRowColor ($saConsumptions | Sort-Object -Property StorageAccountName) -Alternating

$Rpt += Get-HTMLContentTable ($saConsumptionsTableStyle) -Fixed

$Rpt += Get-HtmlContentClose

##

$Rpt += Get-HtmlContentOpen -HeaderText "Total of Standard Storage space allocated on Azure"

$Rpt += Get-HTMLContentText -Heading "Total (GB)" -Detail "$saConsumptionsTotal"

$Rpt += Get-HtmlContentClose

##

if ( $saPremiumUsage -ne $null) {

$Rpt += Get-HtmlContentOpen -HeaderText "Premium Storage Accounts Consumptions (GBs)"

$saPremiumUsageTableStyle = Set-TableRowColor ($saPremiumUsage | Sort-Object -Property StorageAccountName) -Alternating

$Rpt += Get-HTMLContentTable ($saPremiumUsageTableStyle) -Fixed

$Rpt += Get-HtmlContentClose

} ##

$Rpt += Get-HtmlContentOpen -HeaderText "Total of Premium Storage space allocated on Azure"

$Rpt += Get-HTMLContentText -Heading "Total (GB)" -Detail "$saPremiumUsageTotal "

$Rpt += Get-HtmlContentClose

##

$Rpt += Get-HTMLClosePage

$date = Get-Date -Format yyyy.MM.dd.hh.mm

$reportName = $subname + "_" + $date

Write-Host "Output folder is: C:\temp\Billing"

Write-Host "Report file name is : " $reportName

$file = Save-HTMLReport -ReportContent $rpt -ShowReport -ReportPath "C:\temp\Billing" -ReportName $reportName

[/powershell]

Save it

Some comments:

  • Line 7 execute Module-Azure, making available its functions
  • Line 9 invoke Connect-Azure function which is declared in Module-Azure as global
  • From Line 12 to 28 it's checked if ReportHTML is installed
  • Line 42 all Standard Storage Account available in the selected subscription are retrieved
  • From Line 44 to Line 60, VHDs allocated space stored in Standard Storage Account is calculated
  • Line 62 all Premium Storage Account available in the selected subscription are retrieved
  • From Line 64 to Line 95, VHDs allocated space stored in Premium Storage Account is calculated
  • From Line 97 to Line 112, sum of all Standard Storage Accounts and of all Premium Storage Account is calculated
  • From Line 115 to Line 168, report is formatted in HTML using ReportHTML functions
  • Line 174 save report in the default location and open default browser to show it

It's time to run the script and getting some reports !!!

From PowerShell editor or from a PowerShell console, run Generate-AzureReport.ps1

Provide Azure credentials

Select target Azure subscription and click on OK button

Sample of Azure Report

Sample of PowerShell output console

Note:

  • Output folder is the folder path where report has been saved
  • Report file name is report name

Thanks for your patience.  Any feedback is  appreciated