I was asked by a client recently on whether it was possible to use Azure DNS zones with Azure Stack for name resolution, as they could not find any information on how to do it. The answer to that is yes, and is actually my preferred method, whether the Azure Stack region is publicly available, or internally within an enterprise.

Here’s how to do it:

From your Azure Subscription, create an new DNS Zone:

Enter details such as Resource Group, location and most importantly, the name of the Zone. For this, I’m using a domain I own called dmc-tech.uk.

DO NOT create a zone for your Azure Stack deployments that includes the region name, e.g. azst.dmc-tech.uk, as it will complicate matters, such as adding additional regions for your domain, or adding DNS names to your Public IP addresses. I’ll get to how you can have multiple regions within your zone later.

Once the zone has been created, make a note of the Name Servers hosting the records for your zone as you will need the to modify the Name Servers with your Domain registrant.

Typically, they will have a name of:

ns1-<xx>.azure-dns.com
ns1-<xx>.azure-dns.net
ns1-<xx>.azure-dns.org
ns1-<xx>.azure-dns.info

I used UK2.net to register my domain, so this is the screen I used to modify the NS records. Other registrants have similar capabilities:

To test that the records have updated, I run NSLookup to perform a check to make sure they have replicated:

As you can see above, the name servers match those I have set for the DNS Zone in Azure.

The next task is to create some A records in the Azure DNS Zone to match the names of the external DNS name servers of your Azure Stack region. You should get the details for this from your OEM when they have performed the deployment, or you can work it out for yourself as the addresses are taken from the Public VIP network range.

For example, if the Public VIP range is 10.10.1.0/24, the addresses would be 10.10.1.15 and 10.10.1.16. This applies for the current Azure Stack release (1908 at time of writing)

Create the records as follows in the Azure portal:

Within your zone, select +Record Set

enter the hostname. you also include the region name of your Azure Stack instance here, so the form is <hostname>.<regionname>
Ensure the record type is set to A
Enter the IP address of the first external DNS server for your Azure Stack instance
Enter the IP address of the second external DNS server for your Azure Stack instance (You don’t have to do this, but I do it for a belt and braces approach)

Click on Apply to save the record. Repeat the above steps to create another A record for the second external DNS Name server, changing the hostname accordingly.

Once you’ve created your A records, you can now create the NS records for your region. The steps to do this are similar to creating an A record:

Within your zone, select +Record Set

enter the name of you region. In this example it’s azs
Ensure the record type is set to NS
Enter the FQDN of the first external DNS server for your Azure Stack instance. It should match the name of the A record you created previously.
Enter the FQDN of the second external DNS server for your Azure Stack instance

Click on Apply to save the record.

We also need to add a Name Server record for <region>.cloudapp, otherwise, when you assign a DNS name to a public IP address, your client won’t be able to resolve the name. The process is the same as above:

Within your zone, select +Record Set

enter the name of you region.cloudapp. In this example it’s azs.cloudapp
Ensure the record type is set to NS
Enter the FQDN of the first external DNS server for your Azure Stack instance. It should match the name of the A record you created previously.
Enter the FQDN of the second external DNS server for your Azure Stack instance

Click on Apply to save the record.

Once you’ve created your records, you should have something that look similar to this:

To test that your DNS changes have replicated, use NSLookup again. As you can see, the name servers for azs.dmc-tech.uk and azs.cloudapp.dmc-tech.uk resolve to the name servers I’ve set in the Azure DNS zone.

To be clear, your external name servers do not have to be exposed to the internet and as you can see in this example, I’m using addresses within the IANA private network ranges. As long as your corporate DNS servers have forwarders to public DNS zones, it should work. When your internal clients try to access your Azure Stack public endpoints, as long as they can route to your Azure Stack external DNS servers and UDP 53 is allowed via your firewalls, all should be good.

If you want to add additional regions using the same domain, simply follow the process above again :)

I hope that clarifies how you can use this kind of setup within your environment

I recently deployed new Azure Stack integrated system, and despite a few of the usual issues I was expecting (network integration!!!), everything went well up until the point of me needing to syndicate items in to the Marketplace via the admin portal. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#connected-scenario

I could download some of the smaller items successfully, such as the VM extensions, but those that were larger, failed.

Initially, I thought it was a transient network issue, so deleted the failed items from the Marketplace and re-attempted the download, but I had the same problem re-occurred.

Because the admin portal only gives 3 states for Marketplace item (Downloading, Downloaded, or Failed), I wanted to try and determine where the problem lay before calling Support.

To do this, I used the Azure Stack Tools, more specifically: Export-AzSOfflineMarketplaceItem CMDLet. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#disconnected-or-a-partially-connected-scenario. By running in PowerShell, I felt I had more chance in figuring out what was going on by using Verbose logging.

To get more verbose information, I used the azcopy option. When I first started investigating the problem, the version of the tools required AZCopy v7.3/8.1. This required installation via an MSI. However, earlier last week, a new version of the tools was released which uses AZCopy v10 https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10. I much prefer Microsoft’s approach to new releases of this tool as it is a single file, does not require installation, and therefore does not require admin rights.

Here’s a little wrapper script based on that in the documentation to download the Azure Stack tools. It also retrieves azcopy v10 and places it into the tools directory:

# Change directory to the root directory. 
cd \

# Download the tools archive.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 
invoke-webrequest `
  https://github.com/Azure/AzureStack-Tools/archive/master.zip `
  -OutFile master.zip

# Expand the downloaded files.
expand-archive master.zip `
  -DestinationPath . `
  -Force

# Change to the tools directory.
cd AzureStack-Tools-master
# Download azcopy v10
invoke-webrequest `
  https://aka.ms/downloadazcopy-v10-windows `
  -OutFile azcopy.zip

# Expand the downloaded files.
expand-archive azcopy.zip `
  -DestinationPath . `
  -Force

Here’s a script to download marketplace items from Public Azure and then upload it to your stamp:

$Region = "local"   # For ASDK, this is local, change to match your region for integrated deployments
$FQDN = "azurestack.external" # For ASDK, this is azurestack.external
$AzSEnvironmentName = "AzureStackadmin" # Change this if you want more than one Azure Stack registraion on your system

$RegistrationRG = 'AzureStack' # Get this from the AdminPortal / Dashboard / Region / Properties / REGISTRATION RESOURCE GROUP if unsure
$RegistrationUserName = '<user>@<your tenant>.onmicrosoft.com' # User with rights to the Registraion Resource Group

$OperatorUserName = '<operator user name>@<your tenant>' # Username of Operator that is contributor/owner or Default Subscriber Subscription
$OperatorAADTenantName = '<Operator tenant name>' # the AAD tenant Name e.g. <mytenant>.onmicrosoft.com

$mktPlcFolder = "D:\Mkt" # Directory to store downloaded content.  Must exist before running
$azcopypath = "C:\AzureStack-Tools-master\azcopy_windows_amd64_10.2.1\azcopy.exe" 

# Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "https://adminmanagement.$Region.$FQDN" `
    -AzureKeyVaultDnsSuffix adminvault.$Region.$FQDN `
    -AzureKeyVaultServiceEndpointResourceId https://adminvault.$Region.$FQDN

# Set your tenant name
$AuthEndpoint = (Get-AzureRmEnvironment -Name $AzSEnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($OperatorAADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]

#Get the credentials for an identity with permission to the subscription that the stamp is registered to.  sed to download Marketplace item from azure
$RegistrationCreds = get-credential  $RegistrationUserName -Message "Enter Azure Subscription Credentials"
# Get the credentials for an identity that has contributor/owner rights to the Default Provider Subscription.  used to upload Marketplace item to the Stamp
$operatorCreds = Get-Credential -Message "Enter the azure stack operator credential:" -UserName $OperatorUserName

# first, connect to Public Azure...
Add-AzureRmAccount -Credential $RegistrationCreds -Environment AzureCloud
Get-AzureRmSubscription | Select-AzureRmSubscription 

cd C:\AzureStack-Tools-master
Import-Module .\Syndication\AzureStack.MarketplaceSyndication.psm1

# Download the item.  You will be prompted to choose from an Out-Grid window...
Export-AzSOfflineMarketplaceItem -Destination $mktPlcFolder -resourceGroup  $RegistrationRG -AzCopyDownloadThreads 16 -azCopyPath $azcopypath

# Once the download has finished, swich to the Stack admin environment and upload the Marketplace item.
Add-AzureRmAccount -EnvironmentName $AzSEnvironmentName -TenantId $TenantId -Credential $operatorCreds
Import-AzSOfflineMarketplaceItem -origin $mktPlcFolder -AzsCredential $operatorCreds

When you run the script, you’ll be prompted for credentials and then which items you want to download. You can choose multiple items (CTRL and select), but I advise selecting 1 item at a time, as you will be prompted to accept the legal terms and conditions, as well as selecting the download method (azcopy) per choice. You may miss the prompts for subsequent items if the first download takes a while.

Once the selection has been made, you’ll see the following:

Select ‘y’ for both questions, and the download should start.

For the environment I was operating in, downloading via the internet took a while, as there were QoS rules applied.

After a while, I saw the following error:

OK, so I seemed to have the same problem via the portal and with the PowerShell tools. As I was using AZCopy for the download, there are logs, so that was the first port of call for me. The logs are stored in the following directory:

%USERPROFILE%\.azcopy

So I navigated there and opened the latest log file. I found the following towards the end:

I’ve highlighted the key entry that pointed me towards the problem:

‘…the MD5 hash of the data, as we received it, did not match the expected value, as found in the Blob/File Service. This means there is a data integrity error OR another tool has failed to keep the stored hash up to date.’

OK, so I thought this could have been a problem with some inline firewall or web proxy, but then I could open the smaller items, such as the icons associated with the marketplace item, or the manifest json files.

To prove if it was an issue with the environment I was operating in or not, I decided to spin up a Windows Server 2016 VM in Azure and attach a 200GB data disk and run through the same process as above. Thankfully, the downloads were a lot quicker, as would be expected given I was using the Azure Network fabric, but I found that the download failed again, and I saw the same error regarding the MD5 hash. Weird!

I decided to see if there was a way I could circumvent the MD5 hash check to see if I could at least complete the download and get something into the Marketplace so I could test if the item worked or not. This capability is not native within the Export-AzSOfflineMarketplaceItem, but there is a parameter within azcopy to do this: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files#download-files. If I could add --check-md5=NoCheck or --check-md5=LogOnly to the azcopy command within Export-AzSOfflineMarketplaceItem, I could at least test it.

This is actually quite simple to do. by editing C:\AzureStack-Tools-master\Syndication\AzureStack.MarketplaceSyndication.psm1 (replace C:\AzureStack-Tools-master to match path where you have the tools), modify lines 591 & 593 (as of the current version at time of writing) to read:

 & $azCopyPath copy $Source $tmpDestination --recursive --check-md5=LogOnly

It should look like this:

If you already had the AzureStack.MarketplaceSyndication.psm1 loaded, simply close your PowerShell session.

Once I made the changes, I retried the process again, and this time, SUCCESS!

The Marketplace images downloaded and I was able to import them to my stamp, with no issue. I was able to deploy VM’s using these images with no problem.

I’m not sure if there’s an underlying problem with azcopy and the Marketplace content, but at least I managed to figure out a workaround that doesn’t appear to have any detrimental effects, so hopefully it can help out someone else who might have a similar problem.

Using an Azure DNS Zone for name resolution in your Azure Stack region.

Azure Stack Marketplace download issue and how to mitigate