, ,

Simplifying Kubernetes deployments on ADFS Azure Stack systems

, ,

The public preview template for Kubernetes on Azure Stack has been out for a few months now, but the ability/guidance has only been available for a short while to deploy on systems using ADFS as the identity provider. That guidance is here: https://docs.microsoft.com/en-us/azure/azure-stack/user/azure-stack-solution-template-kubernetes-adfs

Feel free to follow the instructions provided, as they do work, but they are fiddly.

Before you start, you have to ensure the following pre-reqs are met before running the template (taken from the doc, but with further comments from me) :

  1. Generate a SSH public and private key pair for the Linux VM’s the template creates. I use PuTTyGen. (Instructions here on generating a key: https://www.ssh.com/ssh/putty/windows/puttygen )

  2. Have a valid tenant subscription where you are at least a contributor. The subscription/region should have enough public IP addresses (at least 2 for core deployment, you’ll need more for services you run on K8s)

  3. Your tenant subscription will need the Key Vault service assigned within the the plan/offer

  4. You’ll need the Kubernetes Cluster marketplace item, or you can just use the ARM template from here: https://github.com/msazurestackworkloads/azurestack-gallery/blob/master/kubernetes/template/DeploymentTemplates/azuredeploy.json


The next part of the doc talks about creating a service principal. This has to be done by an Azure Stack Operator. Currently, the Kubernetes template only supports service principals with certificates for ADFS systems, despite the fact that client secrets was introduced in 1811. Hopefully this will be addressed and supported in a future version, as it will remove the requirement for the certificate and KeyVault.

Once you’ve got the certificate, it needs to be uploaded to a KeyVault within your tenant subscription. The script provided in the doc does this for you, but you need to plug in quite a bit of information and there is the prospect of getting it wrong.

I’ve simplified the process of creating the cert, service principal, creating a key vault and uploading the cert as a secret by producing a script to do the hard work for you. To run it, you need to be an Azure Stack Operator (ability to connect to the ERCS) as well as having access to tenant subscription on the stamp of which you are a contributor.

The script does the following:

  1. Checks if a KeyVault exists on the Azure Stack region with the name you have specified (if it does, it quits)

  2. Creates a self-signed cert on the system you’re running the script on

  3. Connects to the ERCS and creates a service principal using the cert

  4. Exports the cert to a PFX file, with password of your choosing

  5. Connects to Tenant subscription (If you have more than one subscription within the region, it will let you choose which one to deploy to)

  6. Creates a Resource Group, a KeyVault within it and sets access policy to your user account

  7. Uploads the certificate to the KeyVault as a secret

  8. Dumps all the information you need for the template to screen and file

 
Param (
    $ERCS = (Read-Host -Prompt "ERCS"),
    $OutputPath = $ENV:Temp,
    $SubscriptionID,
    [ValidateNotNullOrEmpty()]
    $appNamePrefix = "appSPN",
    [ValidateNotNullOrEmpty()]
    $ResourceGroup = "K8sDemoAdFsRG",
    [ValidateNotNullOrEmpty()]
    $KeyvaultName = "K8sDemoAdFsKV51",
    [ValidateNotNullOrEmpty()]
    $keyVaultSecretName = "K8sSecret",
    [ValidateNotNull()]
    [System.Management.Automation.PSCredential]
    [System.Management.Automation.Credential()]
    $ErcsCredential = (Get-Credential -Message "Enter CloudAdmin Credentials for ERCS"),  
    [ValidateNotNull()]
    [System.Management.Automation.PSCredential]
    [System.Management.Automation.Credential()]
    $cloudCredential = (Get-Credential -Message "Enter Azure Stack Tenant Credentials"),
    [ValidateNotNullOrEmpty()]
    [Security.SecureString]$PfxPassword=(Read-Host "Enter PFX Password" -AsSecureString)

)

[bool]$GeneratePFX = $true

function write-log($logentry){
    Write-output "$logentry" | out-file $detailFile -Append
    Write-output "$logentry" 

}


# Location to write PFX file and log file
$OutputPath = $OutputPath.Trim('\')
if (!(Test-Path $OutputPath -pathType container)) {
	New-Item $OutputPath -type directory -Force
}



# Creating a PSSession to the ERCS PrivilegedEndpoint 
$session = New-PSSession -ComputerName $ERCS -ConfigurationName PrivilegedEndpoint -Credential $ErcsCredential
$AzureStackInfo = Invoke-Command -Session $session -ScriptBlock { get-azurestackstampinformation } 


# For Azure Stack development kit, this value is set to https://management.local.azurestack.external. We will read this from the AzureStackStampInformation output of the ERCS VM. 
$ArmEndpoint = $AzureStackInfo.TenantExternalEndpoints.TenantResourceManager 

# For Azure Stack development kit, this value is set to https://graph.local.azurestack.external/. We will read this from the AzureStackStampInformation output of the ERCS VM. 
$GraphAudience = "https://graph." + $AzureStackInfo.ExternalDomainFQDN + "/" 
# TenantID for the stamp. We will read this from the AzureStackStampInformation output of the ERCS VM. 
$TenantID = $AzureStackInfo.AADTenantID 

# Register an AzureRM environment that targets your Azure Stack instance 
Add-AzureRMEnvironment ` -Name "azurestacktenant" ` -ArmEndpoint $ArmEndpoint 

$location = $AzureStackInfo.RegionName
# Set the GraphEndpointResourceId value 
$AzsEnv = Set-AzureRmEnvironment ` -Name "azurestacktenant" -GraphAudience $GraphAudience -EnableAdfsAuthentication:$true 
$KeyVaultSuffix = $azsEnv.AzureKeyVaultDnsSuffix
$KeyvaultDnsName = "https://" + $KeyvaultName + "." + $KeyVaultSuffix 
$KVSuffix = '/secrets/Secret1?api-version=2016-10-01'
$KVCheckURI = $KeyvaultDnsName + $KVSuffix

# This block of code in untidy, but tests whether the KeyVault namespace exists on the Stamp already (401) or not (404)
try { 
    (Invoke-WebRequest -Uri $KVCheckURI -ErrorAction Stop).BaseResponse
} catch [System.Net.WebException] { 
    # Messy, but we're not using a token to authenticate, just seeing if the name is already in use
    $Status = $_.Exception.Response.StatusCode.value__ 
    If ($Status -eq 404) {
       $stat = "does not exist"
    }
    else
    {
        $stat = "exists already"
    }
    Write-Debug ("KeyVault Namespace {0} {1} in Region {2}" -f $KeyvaultDnsName, $stat, $Location)
    
} 

# Only carry on if the KeyVault namespace doesn't exist on the Stamp 
If ($Status -eq 404) { 
     Write-Debug "Creating Self-signed cert and new Graph APplication..."
     # This produces a self signed cert for testing purposes. It is preferred to use a managed certificate for this. 
  
     if ($GeneratePFX) {
       $cert = New-SelfSignedCertificate -CertStoreLocation "cert:\CurrentUser\My" -Subject "CN=$appNamePrefix" -KeySpec KeyExchange 
       $ServicePrincipal = Invoke-Command -Session $session {New-GraphApplication -Name $args[0] -ClientCertificates $args[1]} -ArgumentList $appNamePrefix,$cert
     }
     else {
       $ServicePrincipal = Invoke-Command -Session $session {New-GraphApplication -Name $args[0] -GenerateClientSecret} -ArgumentList $appNamePrefix
     }

    $session|remove-pssession 

    $SPNName = $ServicePrincipal.ApplicationName
    $PFXFile    = "$OutputPath\$SPNName.pfx"
    $detailfile = "$OutputPath\$SPNName-details.txt"


    write-Log "Client Id          : $($ServicePrincipal.ClientId)"
    if ($GeneratePFX) { write-output "Cert Thumbprint  : $($ServicePrincipal.Thumbprint)"}
    else              { write-output "Client Secret    : $($ServicePrincipal.ClientSecret)"}
    write-Log "Application Name  : $($ServicePrincipal.ApplicationName)"
    write-Log "TenantID          : $TenantID"
    write-Log "ARM EndPoint      : $ArmEndpoint"
    write-Log "Admin Endpoint    : $AdminEndpoint" 
    write-Log "Graph Audience    : $GraphAudience" 



    # Now Export the cert to PFX
    if ($GeneratePFX){
       # enter a password for the PFX file...
       $pw = $PfxPassword
       # Store the cert in the designated output directory
       Export-PfxCertificate -cert $cert -FilePath $PFXFile -Password $pw
       write-Log "PFX Certificate   : $PFXFile" 

    }

    # Connect to the Stamp
    If ($SubscriptionID) {
        $AzsUser = Login-AzureRmAccount  -Environment azurestacktenant -Credential $Cloudcreds -Subscription $subscriptionId
    }
    else
    {
        $AzsUser = Login-AzureRmAccount  -Environment azurestacktenant -Credential $Cloudcreds
        $Subs = Get-AzureRmSubscription
        # Show a list of subs if more than one is available
        If ($Subs.Count -gt 1) {
            $context = $Subs | Out-GridView -PassThru
            Set-AzureRmContext -Subscription $context

        }
    }

    #Get the SID for the user account you've used to connect to the Subscription
    $adfsuserID = $null
   
    try {   
        # using the get-azurermaduser means the script can be used on non-domain joined systems :)
        $adfsuserID = (get-azurermaduser -UserPrincipalName $azsuser.Context.Account.Id).AdfsID 
    } 
    catch {
       
    }
    # This can be used for currently logged in user: 
    <#
    if (-not $adfsuserID) {
        $Filter = "name = '" + $env:USERNAME + "' AND domain = '" + $env:USERDOMAIN + "'"
        $adfsuserID = (Get-WmiObject win32_useraccount -Filter "$Filter").SID
    }
    #>
    
    # Create new Resource group and key vault
    
    New-AzureRmResourceGroup -Name $ResourceGroup -Location $location -Force

    New-AzureRmKeyVault -VaultName $KeyvaultName -ResourceGroupName $ResourceGroup -Location $location -EnabledForTemplateDeployment

    Set-AzureRmKeyVaultAccessPolicy -VaultName $KeyvaultName -ResourceGroupName $ResourceGroup -ObjectId $adfsuserID -BypassObjectIdValidation -PermissionsToKeys all -PermissionsToSecrets all

    #Convert the secure pw to something that can be used
    $BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pw)
    $password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR)

    $certContentInBytes = [io.file]::ReadAllBytes($PFXFile)
    $pfxAsBase64EncodedString = [System.Convert]::ToBase64String($certContentInBytes)
$jsonObject = @"
{
"data": "$pfxAsBase64EncodedString",
"dataType" :"pfx",
"password": "$password"
}
"@
    $jsonObjectBytes = [System.Text.Encoding]::UTF8.GetBytes($jsonObject)
    $jsonEncoded = [System.Convert]::ToBase64String($jsonObjectBytes)
    $secret = ConvertTo-SecureString -String $jsonEncoded -AsPlainText -Force
    $keyVaultSecret = Set-AzureKeyVaultSecret -VaultName $KeyvaultName -Name $keyVaultSecretName -SecretValue $secret

    #Give the new Service Principal Contributor rights to the Subscription
    New-AzureRmRoleAssignment -ApplicationID ($ServicePrincipal.ClientId) -RoleDefinitionName "Contributor" -Scope "/subscriptions/$($context.Id)"

    Write-Log ('')
    Write-Log "Service principal clientId     : $($ServicePrincipal.ClientId)"
    Write-Log "Key vault resource group       : $ResourceGroup "
    Write-Log "Key vault name                 : $KeyvaultName"
    Write-Log "Key vault secret               : $keyVaultSecretName"


    $detailfile
}
else {
    write-Error "Certificate and Keyvault processing halted as KeyVault namespace already exists in this region. Please try another name"
}

When you run the script, you should hopefully see output resembling this:

1_poshoutput.png

I’ve formatted it so that you can copy paste it into the template. I could have created a parameter file, but for my purposes this was fine.

1_k8stemplate.png


For a deeper understanding of whats happening when deploying the template, take a look at Ned Bellavance’s great post here: https://nedinthecloud.com/2019/02/19/azure-stack-kubernetes-cluster-is-not-aks/ .