Crying Cloud

#Azurestack

,

Rotating App Service Certificates on Azure Stack Hub

,

If anyone has seen my previous post about using Lets Encrypt open source certificates with Azure Stack Hub may remember that those certs have a 90 day lifetime. This means that if you do use them, you’ll need to rotate your certificates on a more regular cadence than the normal 1 year with paid for third party cert authorities. For me, this isn’t a problem as it adds only a small amount of overhead to managing my stamps, assisted by helper scripts to generate the certs and some clear documentation on the process for rotating the core infra SSL Certs (found here).

What isn’t so clear is how to rotate the certs for your App Service PaaS infrastructure. I haven’t found any easily findable (is that a word?) reference to it in the online documentation. Not to worry, it’s actually a fairly simple process and I’ve recorded step-by-step how to perform it.

One thing I’ll point out first - you won’t find any alerts regarding impending App Service certificates expiration like you do with the core infrastructure, so you’ll have to make a note in your calendar for a suitable date to renew your certs, or just periodically check the admin portal.

  • From the Admin Portal, open up the App Service blade

  • Open the Secrets blade

  • From here you can check how many days you have left until your certs expire. As you can see in my case I have 11 days, so time to rotate!

  • Selecting the Rotate button brings up a new blade. From here you provide the locations of the Default App Service pfx cert, API pfx, FTP (or publishing) pfx and the SSO (or Identity Application) pfx, along with the corresponding passwords for these files.

  • Once you have entered these details and clicked OK, the Certificate Rotation status window will show you the progress of the task

  • You can navigate from this screen and do other tasks if you want. You can click on the Status button to check on the progress.

What you should see if the details you entered were validated.

Success!

The whole process took between 15-20 minutes. I wasn’t sure as I went and got a coffee!

Just to check that the certificates have actually been updated, I refreshed the Secrets blade and saw that the expiry was now 90 days, per the new Lets Encrypt certs I created.

Hopefully this is of some use to fellow Azure Stack Hub operators who haven’t had to update their App Service Certs yet, and remember, you won’t be alerted if they do expire!

Extending Storage into Azure Stack with iSCSI

If you understand Azure Stack and its storage you may know, Azure Stack storage is highly redundant. Meaning, if you are using NVMe or other flash style storage you can end up consuming very expensive storage with content that may not need that level of redundancy or quality of speed.

Working with Microsoft we have created a pattern to help explore some options for using the iSCSI protocol to mount iSCSI disks into your Azure Stack environment. A simple start is to set up a server that can present iSCSI targets to your IaaS systems and mount iSCSI disk inside your VMs. This means you can use the storage without dealing with share permissions or complex authentication as the storage is secured by IP and Chap authentication. This allows it to be used inside the Azure Stack environment as you see fit. This does open up options for large storage scenarios where you can serve storage up from your existing infrastructure, perhaps being able to leverage your existing investments with SAN or NAS infrastructure.

Below is a high-level overview of how this is implemented.

ExtendStorageIntoAzureStackWithiSCSI.jpg

You can find more details and deployment scripts to help deploy and integrate into your own deployments and environments. There are of course many ways to serve up iSCSI storage this is meant as a stepping stone for you to explore what solutions are right for you and your business.

https://github.com/Azure-Samples/azure-intelligent-edge-patterns/tree/master/storage-iSCSI

We hope this unlocks some value or creates some scenarios to expand the possibilities Azure Stack can offer.

Issue adding additional workers using the administrator portal

I have recently deployed the App Service Update 7 resource provider to an Azure Stack 1908 region and came across a problem when trying to add instances to the web worker tiers via the admin portal, per the instructions here: https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-app-service-add-worker-roles?view=azs-1908#add-additional-workers-using-the-administrator-portal

The problem I came across was when I opened the Roles blade from within the App Service resource provider, trying to select the ScaleSet option for the worker tier was not possible.

I could select the Virtual Machine option if I selected the Controller role:

I checked the release notes for App Service Update 7 and couldn’t find anything that referenced the problem I have: https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-app-service-release-notes-update-seven?view=azs-1908

OK, so I can’t follow the documentation verbatim, but, the workaround is very straightforward. The worker tiers use VM scale sets, so from the admin portal, open up the Virtual machine scale sets admin blade:

Then select the worker tier you wan to modify:

Select the Scaling option and modify the instance count using the slider, or manually enter the number you want. Click Save to start the provisioning process.

Open the App Service / Worker Tiers blade; it will show the instance count has now increased:

It will take a while for the available instances to increment, so be patient.

You can, however, see that the instance count has increased if you select the Roles blade

Selecting a Worker instance will show you a little more detail. The example below, shows that 2 instances are installing:

Eventually, you should hopefully see something like the following:

Of course, you can use the PowerShell method, and that works out of the box, but I’ve modified the script in the document to be more Integrated system friendly :)

##### Scale out the AppService Role instances ######

 # Set context to AzureStack admin.
 Login-AzureRmAccount -EnvironmentName AzureStackAdmin

 $RegionName = 'local' # Change this to reflect the region.   ASDK is 'local'

 ## Name of the Resource group where AppService is deployed.
 $AppServiceResourceGroupName = "AppService.$RegionName"

 ## Name of the ScaleSet : e.g. FrontEndsScaleSet, ManagementServersScaleSet, PublishersScaleSet , LargeWorkerTierScaleSet,      MediumWorkerTierScaleSet, SmallWorkerTierScaleSet, SharedWorkerTierScaleSet
 $ScaleSetName = "SmallWorkerTierScaleSet"

 ## TotalCapacity is sum of the instances needed at the end of operation. 
 ## e.g. if your VMSS has 1 instance(s) currently and you need 1 more the TotalCapacity should be set to 2
 $TotalCapacity = 2  

 # Get current scale set
 $vmss = Get-AzureRmVmss -ResourceGroupName $AppServiceResourceGroupName -VMScaleSetName $ScaleSetName

 # Set and update the capacity
 $vmss.sku.capacity = $TotalCapacity
 Update-AzureRmVmss -ResourceGroupName $AppServiceResourceGroupName -Name $ScaleSetName -VirtualMachineScaleSet $vmss

Azure Stack Marketplace download issue and how to mitigate

I recently deployed new Azure Stack integrated system, and despite a few of the usual issues I was expecting (network integration!!!), everything went well up until the point of me needing to syndicate items in to the Marketplace via the admin portal. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#connected-scenario

 

I could download some of the smaller items successfully, such as the VM extensions, but those that were larger, failed.

Initially, I thought it was a transient network issue, so deleted the failed items from the Marketplace and re-attempted the download, but I had the same problem re-occurred.

 

Because the admin portal only gives 3 states for Marketplace item (Downloading, Downloaded, or Failed), I wanted to try and determine where the problem lay before calling Support.

To do this, I used the Azure Stack Tools, more specifically: Export-AzSOfflineMarketplaceItem CMDLet. https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-download-azure-marketplace-item?view=azs-1908#disconnected-or-a-partially-connected-scenario. By running in PowerShell, I felt I had more chance in figuring out what was going on by using Verbose logging.

 To get more verbose information, I used the azcopy option. When I first started investigating the problem, the version of the tools required AZCopy v7.3/8.1. This required installation via an MSI. However, earlier last week, a new version of the tools was released which uses AZCopy v10 https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10. I much prefer Microsoft’s approach to new releases of this tool as it is a single file, does not require installation, and therefore does not require admin rights.  

 Here’s a little wrapper script based on that in the documentation to download the Azure Stack tools. It also retrieves azcopy v10 and places it into the tools directory:

# Change directory to the root directory. 
cd \

# Download the tools archive.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 
invoke-webrequest `
  https://github.com/Azure/AzureStack-Tools/archive/master.zip `
  -OutFile master.zip

# Expand the downloaded files.
expand-archive master.zip `
  -DestinationPath . `
  -Force

# Change to the tools directory.
cd AzureStack-Tools-master
# Download azcopy v10
invoke-webrequest `
  https://aka.ms/downloadazcopy-v10-windows `
  -OutFile azcopy.zip

# Expand the downloaded files.
expand-archive azcopy.zip `
  -DestinationPath . `
  -Force

Here’s a script to download marketplace items from Public Azure and then upload it to your stamp:

$Region = "local"   # For ASDK, this is local, change to match your region for integrated deployments
$FQDN = "azurestack.external" # For ASDK, this is azurestack.external
$AzSEnvironmentName = "AzureStackadmin" # Change this if you want more than one Azure Stack registraion on your system

$RegistrationRG = 'AzureStack' # Get this from the AdminPortal / Dashboard / Region / Properties / REGISTRATION RESOURCE GROUP if unsure
$RegistrationUserName = '<user>@<your tenant>.onmicrosoft.com' # User with rights to the Registraion Resource Group

$OperatorUserName = '<operator user name>@<your tenant>' # Username of Operator that is contributor/owner or Default Subscriber Subscription
$OperatorAADTenantName = '<Operator tenant name>' # the AAD tenant Name e.g. <mytenant>.onmicrosoft.com

$mktPlcFolder = "D:\Mkt" # Directory to store downloaded content.  Must exist before running
$azcopypath = "C:\AzureStack-Tools-master\azcopy_windows_amd64_10.2.1\azcopy.exe" 

# Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name "AzureStackAdmin" -ArmEndpoint "https://adminmanagement.$Region.$FQDN" `
    -AzureKeyVaultDnsSuffix adminvault.$Region.$FQDN `
    -AzureKeyVaultServiceEndpointResourceId https://adminvault.$Region.$FQDN

# Set your tenant name
$AuthEndpoint = (Get-AzureRmEnvironment -Name $AzSEnvironmentName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($OperatorAADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]

#Get the credentials for an identity with permission to the subscription that the stamp is registered to.  sed to download Marketplace item from azure
$RegistrationCreds = get-credential  $RegistrationUserName -Message "Enter Azure Subscription Credentials"
# Get the credentials for an identity that has contributor/owner rights to the Default Provider Subscription.  used to upload Marketplace item to the Stamp
$operatorCreds = Get-Credential -Message "Enter the azure stack operator credential:" -UserName $OperatorUserName

# first, connect to Public Azure...
Add-AzureRmAccount -Credential $RegistrationCreds -Environment AzureCloud
Get-AzureRmSubscription | Select-AzureRmSubscription 

cd C:\AzureStack-Tools-master
Import-Module .\Syndication\AzureStack.MarketplaceSyndication.psm1

# Download the item.  You will be prompted to choose from an Out-Grid window...
Export-AzSOfflineMarketplaceItem -Destination $mktPlcFolder -resourceGroup  $RegistrationRG -AzCopyDownloadThreads 16 -azCopyPath $azcopypath

# Once the download has finished, swich to the Stack admin environment and upload the Marketplace item.
Add-AzureRmAccount -EnvironmentName $AzSEnvironmentName -TenantId $TenantId -Credential $operatorCreds
Import-AzSOfflineMarketplaceItem -origin $mktPlcFolder -AzsCredential $operatorCreds

When you run the script, you’ll be prompted for credentials and then which items you want to download. You can choose multiple items (CTRL and select), but I advise selecting 1 item at a time, as you will be prompted to accept the legal terms and conditions, as well as selecting the download method (azcopy) per choice. You may miss the prompts for subsequent items if the first download takes a while.

Once the selection has been made, you’ll see the following:

Select ‘y’ for both questions, and the download should start.

For the environment I was operating in, downloading via the internet took a while, as there were QoS rules applied.

After a while, I saw the following error:

OK, so I seemed to have the same problem via the portal and with the PowerShell tools. As I was using AZCopy for the download, there are logs, so that was the first port of call for me. The logs are stored in the following directory:

%USERPROFILE%\.azcopy

So I navigated there and opened the latest log file. I found the following towards the end:

I’ve highlighted the key entry that pointed me towards the problem:

‘…the MD5 hash of the data, as we received it, did not match the expected value, as found in the Blob/File Service. This means there is a data integrity error OR another tool has failed to keep the stored hash up to date.’

OK, so I thought this could have been a problem with some inline firewall or web proxy, but then I could open the smaller items, such as the icons associated with the marketplace item, or the manifest json files.

To prove if it was an issue with the environment I was operating in or not, I decided to spin up a Windows Server 2016 VM in Azure and attach a 200GB data disk and run through the same process as above. Thankfully, the downloads were a lot quicker, as would be expected given I was using the Azure Network fabric, but I found that the download failed again, and I saw the same error regarding the MD5 hash. Weird!

I decided to see if there was a way I could circumvent the MD5 hash check to see if I could at least complete the download and get something into the Marketplace so I could test if the item worked or not. This capability is not native within the Export-AzSOfflineMarketplaceItem, but there is a parameter within azcopy to do this: https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-files#download-files. If I could add --check-md5=NoCheck or --check-md5=LogOnly to the azcopy command within Export-AzSOfflineMarketplaceItem, I could at least test it.

This is actually quite simple to do. by editing C:\AzureStack-Tools-master\Syndication\AzureStack.MarketplaceSyndication.psm1 (replace C:\AzureStack-Tools-master to match path where you have the tools), modify lines 591 & 593 (as of the current version at time of writing) to read:

 & $azCopyPath copy $Source $tmpDestination --recursive --check-md5=LogOnly

It should look like this:

If you already had the AzureStack.MarketplaceSyndication.psm1 loaded, simply close your PowerShell session.

Once I made the changes, I retried the process again, and this time, SUCCESS!

The Marketplace images downloaded and I was able to import them to my stamp, with no issue. I was able to deploy VM’s using these images with no problem.

I’m not sure if there’s an underlying problem with azcopy and the Marketplace content, but at least I managed to figure out a workaround that doesn’t appear to have any detrimental effects, so hopefully it can help out someone else who might have a similar problem.

Azure Stack Update 1905 - Additional Resource Requirements

As I’ve been away on holiday (vacation) I missed out on a fairly crucial piece of information from the release notes for Integrated systems.

From https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-release-notes-1905:

1905-increaseresourceconsumption.png

How many people picked up on this?

So, be prepared after you’ve run the update to lose some of the capacity that was previously available to tenants.

I confess to not having any insight into how the Infrastructure Ring will work, and was at first dubious to the claim of reducing overall consumption at first, but my guess is that on each hardware node there will be a VM which will be running Service Fabric, and from within there, they will run containerized versions of the XRP, ACS (Azure Consistent Storage), WAS (Admin management endpoint & portal), WASP (User management endpoint & portal). I’m not sure if the Network Controllers, Gateway or SLB’s could move to the containerized platform.

Doing some rough calculations by looking at the Infrastructure Roles Blade in the Admin Portal, I calculated that currently the VM’s hosting the roles I guessed would be moving currently consume 102 GB and 36 vCPU.

Example of where to find out the VM sizes of infra roles in the Admin Portal:

1905-rolevms.png

So, in reality it looks like vCPU consumption will in fact increase, but Memory decrease, and you know what, I’m OK with that. Memory is the most precious resource in my opinion, as it is not over-committed. You’re likely to run out of that before Compute.

Hopefully we can see the benefits of the new architecture soon.