Developing an immutable infrastructure for your Azure DevOps pipelines – Part 1

Our scope is to develop an Azure DevOps CICD Pipeline that will create a Windows Server 2016 custom image based off the Marketplace image gallery which is according to company standards and is running our application.

  1.  Deploy a vanilla Windows Server 2016 VM using Packer
  2.  Bake in company required software(for the sake of simplicity of this exercise we will automatically download and install BgInfo from Sysinternals)
  3. Install IIS and deploy our App
  4. Sysprep the VM and convert it into an Azure custom image
  5. Use the custom image to deploy 3 VMs

Why an immutable infrastructure?

  • Aligns with software development cycles
  • Fast replacement of the components for every deployment, ensures exact alignment between multiple systems according to company standards
  • Best suitable for cloud and virtual environments
  • Fast delivery of new workloads when load increase is expected
  • By discarding the infrastructure when is not needed/used it helps lowering the costs

Setting up the connection to your Azure Subscription

In order for the release pipeline to be able to connect to your Azure subscription you will need to create a service principal(AAD Application). The easiest way to do this is by going to the Project Settings, Service Connections and click on New service connection. Select Azure Resource Manager. Once your service principal is created click on Manage Service Principal and store the ApplicationID. Click on setting and generate a new Key. Make sure you store it before closing the window.

Setting up the repository

In your Azure DevOps project repository add a new json file that will be used as a configuration file by Packer.

{
    "variables": {
    "azure_tenant_id": "{{env `ARM_TENANT_ID`}}",
    "azure_client_id": "{{env `ARM_CLIENT_ID`}}",
    "azure_client_secret": "{{env `ARM_CLIENT_SECRET`}}",
    "azure_subscription_id": "{{env `ARM_SUBSCRIPTION_ID`}}"
  },	
  "builders": [
        {
            "name": "Azure",
            "type": "azure-arm",
      "client_id": "{{user `azure_client_id`}}",
      "client_secret": "{{user `azure_client_secret`}}",
      "subscription_id": "{{user `azure_subscription_id`}}",
            "tenant_id": "{{user `azure_tenant_id`}}",
            "managed_image_resource_group_name": "cc-demo-devops-pipeline",
      "managed_image_name": "CC-Windows2016-AppBASE",
            "os_type": "Windows",
            "image_publisher": "MicrosoftWindowsServer",
            "image_offer": "WindowsServer",
            "image_sku": "2016-Datacenter",
            "communicator": "winrm",
            "winrm_use_ssl": "true",
            "winrm_insecure": "true",
            "winrm_timeout": "3m",
            "winrm_username": "packer",
            "location": "eastus",
            "vm_size": "Standard_DS3_V2"
        }
    ],
    "provisioners": [
        {
            "type": "windows-restart",
            "restart_check_command": "powershell -command \"& {Write-Output 'restarted.'}\""
        },
        {
            "type": "powershell",
            "inline": [
                "Invoke-WebRequest -uri 'https://download.sysinternals.com/files/BGInfo.zip' -OutFile 'c:\\BGInfo.zip'",
                "Expand-Archive -Path 'c:\\BGInfo.zip' -DestinationPath 'C:\\'",
                "& 'C:\\BGInfo.exe' --% /silent",
                "& $env:SystemRoot\\System32\\Sysprep\\Sysprep.exe /oobe /generalize /quiet /quit",
                "while($true) { $imageState = Get-ItemProperty HKLM:\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup\\State | Select ImageState; if($imageState.ImageState -ne 'IMAGE_STATE_GENERALIZE_RESEAL_TO_OOBE') { Write-Output $imageState.ImageState; Start-Sleep -s 10  } else { break } }"
            ]
        }
    ]
}

Setting up the pipeline

Navigate to Releases and click on New Pipeline and Start with an empty job. Give the first stage a name. Mine will be named Image Build.

Connect your pipeline to your Azure DevOps repository by clicking on the Add an artifact button, in this example I am using a TFVC type repository.

 

 

Click on Tasks and then on the plus sign next to your agent then search for “Build Machine Image” and then click Add.

Configure the task as follows:

  • Version: 1.* – this is required as lower versions of Packer do not support Managed disks.
  • Packer Template – User Provided
  • Packer template location – click on the three dots and browse your repository to select the uploaded json
  • Image URL or name – IMAGEURI

Because the uploaded configuration Packer json is using variables, we will have to add them the the release definition. Click on Variables on the horizontal menu and then start adding the following variables:

  • ARM_CLIENT_ID – ApplicationID of the service principal
  • ARM_CLIENT_SECRET – Service principal`s key
  • ARM_SUBSCRIPTION_ID – Azure Subscription ID where the deployment should take place
  • ARM_TENANT_ID – Your organizations Azure AD ID
  • IMAGEURI – Leave empty

Let`s give it a spin!

We are now in a position were we can test out the image creation process. Go to Releases, select your newly created pipeline and Create a release.

If you check your subscription during the deployment you will see a new temporary Resource Group was automatically created by Packer.

This is where the temporary machine is being created.Packer will deploy the deploy these temporary resources: KeyVault, VNET, Virtual Machine, Disk and Network Interface. It will then connect over WinRM to the VM to deploy the specified configuration from our JSON File.

VM is now successfully deployed and syspreped so Packer will capture the image, save it to the specified resource group and cleanup all the temporary objects it created.

2018-11-16T19:31:19.0581380Z ==> Azure: Powering off machine ...
2018-11-16T19:31:19.0581447Z 
2018-11-16T19:31:19.0584764Z ==> Azure:  -> ResourceGroupName : 'packer-Resource-Group-zmk5m9nwqd'
2018-11-16T19:31:19.0585526Z 
2018-11-16T19:31:19.0597321Z ==> Azure:  -> ComputeName       : 'pkrvmzmk5m9nwqd'
2018-11-16T19:31:19.0597520Z 
2018-11-16T19:32:19.2704748Z ==> Azure: Capturing image ...
2018-11-16T19:32:19.2709478Z 
2018-11-16T19:32:19.2711493Z ==> Azure:  -> Compute ResourceGroupName : 'packer-Resource-Group-zmk5m9nwqd'
2018-11-16T19:32:19.2711543Z 
2018-11-16T19:32:19.2733084Z ==> Azure:  -> Compute Name              : 'pkrvmzmk5m9nwqd'
2018-11-16T19:32:19.2738129Z 
2018-11-16T19:32:19.2738611Z ==> Azure:  -> Compute Location          : 'eastus'
2018-11-16T19:32:19.2738678Z 
2018-11-16T19:32:19.3814690Z ==> Azure:  -> Image ResourceGroupName   : 'cc-demo-devops-pipeline'
2018-11-16T19:32:19.3814923Z 
2018-11-16T19:32:19.3815634Z ==> Azure:  -> Image Name                : 'CC-Windows2016-AppBASE'
2018-11-16T19:32:19.3815663Z 
2018-11-16T19:32:19.3822636Z ==> Azure:  -> Image Location            : 'westeurope'
2018-11-16T19:32:19.3822712Z 
2018-11-16T19:33:21.9039265Z ==> Azure: Deleting resource group ...
2018-11-16T19:33:21.9039604Z 
2018-11-16T19:33:21.9040260Z ==> Azure:  -> ResourceGroupName : 'packer-Resource-Group-zmk5m9nwqd'
2018-11-16T19:33:21.9040295Z 
2018-11-16T19:33:21.9054703Z ==> Azure: 
2018-11-16T19:33:21.9055088Z 
2018-11-16T19:33:21.9057210Z ==> Azure: The resource group was created by Packer, deleting ...
2018-11-16T19:33:21.9057470Z 
2018-11-16T19:38:08.9125811Z ==> Azure: Deleting the temporary OS disk ...
2018-11-16T19:38:08.9126670Z 
2018-11-16T19:38:08.9132398Z ==> Azure:  -> OS Disk : skipping, managed disk was used...
2018-11-16T19:38:08.9132511Z 
2018-11-16T19:38:08.9132846Z ==> Azure: Deleting the temporary Additional disk ...
2018-11-16T19:38:08.9132915Z 
2018-11-16T19:38:08.9142808Z ==> Azure:  -> Additional Disk : skipping, managed disk was used...
2018-11-16T19:38:08.9142870Z 
2018-11-16T19:38:08.9143313Z Build 'Azure' finished.
2018-11-16T19:38:08.9143342Z 
2018-11-16T19:38:08.9173918Z 
2018-11-16T19:38:08.9174387Z 
2018-11-16T19:38:08.9176172Z ==> Builds finished. The artifacts of successful builds are:
2018-11-16T19:38:08.9177124Z 
2018-11-16T19:38:08.9202559Z --> Azure: Azure.ResourceManagement.VMImage:
2018-11-16T19:38:08.9205680Z 
2018-11-16T19:38:08.9205901Z 
2018-11-16T19:38:08.9206664Z ManagedImageResourceGroupName: cc-demo-devops-pipeline
2018-11-16T19:38:08.9206709Z ManagedImageName: CC-Windows2016-AppBASE
2018-11-16T19:38:08.9206747Z ManagedImageLocation: westeurope
2018-11-16T19:38:08.9206770Z 
2018-11-16T19:38:08.9206828Z 
2018-11-16T19:38:09.2118727Z packer build completed.
2018-11-16T19:38:09.2504947Z ##[section]Finishing: Build immutable image

In Part two of this series we will extend the JSON to deploy IIS and a basic App into our image. Stay tuned!

 

 

Automating the creation of VSTS Tasks with Microsoft Flow and Google Sheets

When it comes to the repetitive creation of VSTS Tasks things can get a little bit complicated and time consuming especially if there are a lot of tasks to create. With this first blog post we aim to make our lives a little more easier and win some time to focus on more important activities.

First step will be to create a New Google Sheets file. Make sure you add the table headers on the first line like it the example below.

Start by creating a flow from blank by logging into Microsoft Flow and then going to My Flows tab. Select the trigger that suits you best, for this example we are going to use a button.

Start constructing your flow by adding the required actions. We are going to save our task list in a Google Sheets file. In order to get the data we have to use a Google Sheets – Get Rows action. Once you add the action you will be required to create a new Google connection if one was not already created.

Select your Google Sheets file created at the beginning of this guide.

The next step is an applied to each loop. Hit Next Step > More and then “Add an apply to each”

The input value for this loop should be the Records output value from the Get Rows action.

Add a new Visual Studio Team Services action inside the loop and create a new connection with your VSTS Account.

Start customizing the action using the dynamic content provided by the Google Sheets file by maching the fields with the Sheets column titles.

At the end your action should look like this:

Update your flow and do a test run.

If the flow was executed successfully you should see all the steps as completed.Now go to your VSTS account project and check the creation of the Workitems.