
Sitecore 9 is moving towards a micro-service based architecture and is designed with the cloud in mind. The community has picked this development up by creating Docker images for Sitecore. This works great for local development, but it still has some challenges when deploying to the cloud. This changed with the recent introduction of Windows support for Kubernetes (K8S) and Azure Kubernetes Service (AKS) support for Windows containers. 


This article introduces a Kubernetes setup, using Sitecore XC 9.1 as showcase, that allows you to deploy to AKS.

# Sitecore XC 9.1 topology
Let's first have a look at the Sitecore XC 9.1 topology. 
The below diagram gives an overview of all the services and databases for a basic XC 9.1 setup.

![](/sitecore_xc91_topology.png)

These services are mapped to several Docker images, this mapping is as follows;

- Content Management & Content Delivery are combined in a **Sitecore** Docker image
- Business Tools and the Commerce Engine roles are grouped in a **Commerce** Docker image
- The Identity Server has its own Docker image, named **Identity**
- XConnect and its Processing part are placed in a **XConnect** Docker image
- All SQL database are present in a **Mssql** image
- All Solr cores are placed in a **Solr** image

These 6 Docker images form the basis for running the containerized Sitecore XC system.
Build the images using the instructions provided in your Github repository of choice, ie. [sitecore/docker-images](https://github.com/Sitecore/docker-images) or [avivasolutionsnl/sitecore-docker](https://github.com/avivasolutionsnl/sitecore-docker).
After building the images push them to a Docker Registry, e.g. Azure Container Registry, such that they can be pulled and used to create containers when deploying to AKS.

# Azure Kubernetes Service
Now that we have a basic overview of a containerized Sitecore XC setup, lets look at Azure Kubernetes Service (AKS).
We run two types of nodes; a Linux and Windows node. The Linux node is used to run the Kubernetes services, and the Windows node is used for running the Sitecore Windows containers.
A basic AKS cluster and its infrastructure is depicted below.

![](/docker_deployment_variants.png)

At the bottom we have the infrastructure for our cluster. It consists of a Linux and Windows VM connected to eachother using a virtual network.

On top of this infrastructure the Kubernetes cluster is created.
The cluster consists of a master Linux node and a (slave) Windows node. Each node runs a number of default services;

- *kubelet*, which is the primary node agent
- *k-proxy*, the network proxy that e.g. handles communication between nodes
- *Docker* Moby as container runtime

On the master node some additional services are running;

- *api*, which is the Kubernetes server
- *etcd*, this is the Kubernetes configuration store
- *scheduler*, the service in charge of placing Pods in the cluster

# Setup AKS cluster
With this understanding of an AKS cluster its time to get hands-on and setup an empty hybrid (Linux & Windows nodes) Kubernetes cluster.
After installing the Azure CLI tooling;

1. Create a resource group

    ```
    PS> az group create --name $ResourceGroup --location $Location
    ```

2. Create a Kubernetes cluster with one Linux node (and already specify the Windows credentials needed to run Windows containers)

    ```
    PS> az aks create --resource-group $ResourceGroup `
        --name $ClusterName `
        --node-count 1 `
        --enable-addons monitoring `
        --kubernetes-version 1.14.5 `
        --generate-ssh-keys `
        --windows-admin-password $WindowsPassword `
        --windows-admin-username $WindowsAdminUser `
        --enable-vmss `
        --network-plugin azure
    ```

3. Add a Windows Server node to the cluster

    ```
    PS> az aks nodepool add --resource-group $ResourceGroup `
        --cluster-name $ClusterName `
        --os-type Windows `
        --name npwin `
        --node-count 1 `
        --kubernetes-version 1.14.5
    ```

That's basically all to get a hybrid AKS cluster up and running! 
More detailed steps, e.g. how to connect ACR to your cluster, are described [here](https://github.com/joostmeijles/xc9-k8s/blob/master/k8s/README.md).

The movie below recaps the steps above and shows some basic commands to inspect your cluster.

<iframe width="560" height="315"
src="https://www.youtube.com/embed/tZUqtDLOSs8" 
frameborder="0" 
allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" 
allowfullscreen/>

# Deploy Solr to AKS
Now that we have an empty AKS cluster up and running, lets see what it takes to deploy a single service.
We choose Solr as use-case as its a backend service with no dependent services. It requires some persistent storage, which is later required by more services. And it does not require any special configuration (e.g. runs HTTP out-of-the-box).

The Kubernetes YAML spec [file](https://github.com/joostmeijles/xc9-k8s/blob/master/xc9/solr.yaml) for Solr looks as follows:
```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: solr-deployment
  labels:
    app: solr
spec:
  replicas: 1
  selector:
    matchLabels:
      app: solr
  template:
    metadata:
      labels:
        app: solr
    spec:
      nodeSelector:
        "beta.kubernetes.io/os": windows # Run on a Windows Node
      containers:
      - name: solr
        image: avivasolutionsnl.azurecr.io/sitecore-xc-solr:9.1.0-20190904
        ports:
          - containerPort: 8983
        volumeMounts:
          - mountPath: "/Data"
            name: volume
      volumes:
        - name: volume
          persistentVolumeClaim:
            claimName: solr-managed-disk
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: solr-managed-disk
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: managed-premium
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
  name: solr
spec:
  ports:
  - protocol: TCP
    port: 8983
  selector:
    app: solr
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: solr
  annotations:
    kubernetes.io/ingress.class: nginx # Use Nginx as Ingress controller
    kubernetes.io/tls-acme: 'true' # Auto-generate SSL certificates and off-load SSL
spec:
  tls:
  - hosts:
    - solr.xc9-k8s.rocks
    secretName: tls-secret-solr
  rules:
  - host: solr.xc9-k8s.rocks
    http:
      paths:
      - backend:
          serviceName: solr
          servicePort: 8983
        path: /
```

As you can see it consists of 4 parts;

- Deployment
- PersistentVolumeClaim
- Service
- Ingress

The *Deployment* part defines which Docker image to use (it uses the XC Solr Docker image from the Aviva ACR) and that this Pod should run on a Windows node, and it mounts persistent volume into the `/Data` path.

Next the *PersistentVolumeClaim* describes to allocate 5 Gi of Azure `managed-premium` storage.

There is a *Service* spec that abstracts the Solr Deployment into a network service. A *Service* for example enables load-balancing over multiple Pods.

Finally there is an *Ingress* spec that defines to use Nginx as Ingress controller, auto-generates SSL certificates and perform SSL-offloading, uses `solr.xc9-k8s.rocks` as DNS name (and configure this automatically as Ingress route), and finally it maps incoming HTTP requests to the service named `solr` on port `8983`.

The desired stated described in the K8S YAML spec can be applied as follows:
```
PS> kubectl apply -f ./xc9/solr.yaml
```

After the persistent storage is allocated, this takes a little while, the Pod is started and you should be able to browse to https://solr.xc9-k8s.rocks 

Watch below movie to see how applying the K8S Solr spec works out in practice:

<iframe width="560" height="315"
src="https://www.youtube.com/embed/6oRDk36usRw" 
frameborder="0" 
allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" 
allowfullscreen/>

# Deploy Sitecore XC 9.1 to AKS
Next step is to deploy the full Sitecore XC 9.1 setup. This involves basically the same steps as shown for Solr, plus some configuration for;

- *Sitecore*, configure the site settings
- *Identity*, configure CORS settings, apply a small patch to enable reverse-proxy support see [here](https://github.com/joostmeijles/Sitecore.Identity.ProxySupport)
- *Commerce*, configure CORS settings

> All K8S YAML spec files for Sitecore XC 9.1 can be found [here](https://github.com/joostmeijles/xc9-k8s/tree/master/xc9).

Besides this we will need to supply a Sitecore license file. We will use a Kubernetes secret, which is a special type of config, to mount the license file in the containers.

To create a K8S secret from a `license.xml` file do:
```
PS> kubectl create secret generic sitecore-license --from-file=license.xml
```

This creates a secret named `sitecore-license` and uploads `license.xml` into it.

To verify the contents of `sitecore-license`:
```
PS> kubectl describe secrets sitecore-license
```

Now that we a secret available we can mount it into the Sitecore, Identity, and XConnnect containers by adding the following snippet to the K8S YAML spec files:
```
...
    container:
        ...
        volumeMounts:
            - mountPath: "/license"
                name: license
                readOnly: true
    volumes:
        - name: license
            secret:
                secretName: sitecore-license
```

You can find all spec files [here](https://github.com/joostmeijles/xc9-k8s/blob/master/xc9/).

Now everything is set up to, apply all XC 9.1 spec files (located in the `xc9` directory) in once:
```
PS> kubectl apply -f ./xc9
```

Wait a bit till all Pods are up and running. You can monitor the progress using:
```
PS> kubectl get pods -w
```

In the meantime you can also inspect the Ingress controller by performing:
```
PS> kubectl get ing
```

Once all Pods are up and running e.g. navigate to https://www.xc9-k8s.rocks to verify that everything works.

A fully detailed description of what to configure can be found [here](https://github.com/joostmeijles/xc9-k8s/blob/master/xc9/README.md).

Want a visual impression of how it all works? Watch the following movie..

<iframe width="560" height="315"
src="https://www.youtube.com/embed/YBYQvZKGqEo" 
frameborder="0" 
allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" 
allowfullscreen/>

# Scale
Till now we have seen how to create a cluster, deploy a service, provision storage, and configure a secret.
This all works pretty smooth with Kubernetes, but the real power of K8S is best shown by its scaling features. Kubernetes allows you to horizontally scale the number of Nodes or Pods.

In this example we will look at how to use the Horizontal Pod Autoscaler (hpa) to scale the number of Commerce Pods.
We plan to scale the number of Commerce Pods from 1 to 3 depending on the CPU load.

First thing is to specify the resources to allocate per Pod. For the Commerce Pod will apply the following:
```
containers:
    - name: commerce
    resources:
        requests:
            cpu: 500m # 50% of 1 vCPU
        limits:
            cpu: 500m
```
Above states that the container requests at least *and* at most 500 milli CPU (i.e. 50% of 1 vCPU).

Next we will add a `HorizontalPodAutoscaler` spec to the Commerce YAML file:
```
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: commerce
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: commerce-deployment
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50 # Target is 50% of the requested CPU, i.e. 50% of 500 milli CPU in this case
```
Above spec requests to add a Pod when the CPU load for the allocated Pods, which is initially the minimum of 1, goes above 50% of the requested 500 milli CPU.
Note that CPU is just one of the metric types you can choose.

To apply load to the system we will perform a simple load-test using [K6 Loadimpact](https://github.com/loadimpact/k6). Finally we need an actual webshop to apply the load to.
We will use the [Mercury demo webshop](https://github.com/avivasolutionsnl/mercury-demo) for this. It is the easiest solution since its offers a pre-configured Dockerized Sitecore XC 9.1 webshop.

> [Mercury](https://mercury-ecommerce.com/) is a Sitecore Commerce accelerator.

Lets spin up the Mercury Demo webshop (find all K8S files for it [here](https://github.com/joostmeijles/xc9-k8s/tree/master/scale)):
```
PS> kubectl apply -f ./scale
```

Once the system is up and running, query the Horizontal Pod Autoscaler as follows:
```
PS> kubectl get hpa
```

It will indicate the average load that it measured for the Commerce Pod(s).

> After start up there is possibly already more than one Commerce Pod running. Wait a bit till it cools down.

As load-test we will add a single product many times to the cart. The K6 script looks as follows (find it [here](https://github.com/joostmeijles/xc9-k8s/blob/master/scale/loadtest.js)):
```
import { check } from "k6";
import http from "k6/http";

export default function() {
    let payload = JSON.stringify({catalogName: "MercuryFood", productID: "4757660", quantity: null, variantDisplayValue: null});
    var params =  { headers: { "Content-Type": "application/json" } }
    let res = http.post("https://mercury-www.xc9-k8s.rocks/mercury/checkout/cart/add", payload, params);
    check(res, {
        "is status 204": (r) => r.status === 204
    });
};

```

To run this script for 60 seconds and 100 virtual users:
```
PS> docker run -i loadimpact/k6 run -u 100 -d 60s -< loadtest.js
```

Now inspect the Horizontal Pod Autoscaler:
```
PS> kubectl get hpa
```
If everything went well you will see that the load goes well beyond the configured average of 50%.
Due to this the HPA will allocate more Pods.

You can get a full description of the HPA its actions by:
```
PS> kubectl describe hpa
```

Note that the autoscaler by default queries every 15 seconds for metrics. When needed it will upscale the number
of Pods. Downscaling Pods will happen after a (default) cool down period of 5 minutes.

You can find an even more detailed description of above steps in [this README](https://github.com/joostmeijles/xc9-k8s/blob/master/scale/README.md).

See Horizontal Pod Autoscaling in action:
<iframe width="560" height="315"
src="https://www.youtube.com/embed/HxroMFxF3p0" 
frameborder="0" 
allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" 
allowfullscreen/>

# Conclusion
Azure Kubernetes Service is the easiest way to deploy your Sitecore Docker containers to the cloud. 
Other options like Docker Swarm and Docker Desktop using VMs have less features, e.g. no automatic storage provisioning, and require more set up work and maintenance.

Creating service deployment specs and an AKS cluster can be done in under one hour, which is a good indication of the convenience AKS brings you.

The shown Sitecore XC 9.1 AKS setup is meant for development, until:

- it has centralized logging, e.g. Azure App Insights configured
- uses an Azure SQL managed database; its no good idea to run your own database containers in production
- AKS for Windows out of preview and thus generally available.

Try it for yourself: https://github.com/joostmeijles/xc9-k8s


XC9 + K8S + AKS = Bingo?! 🌟

 
Since a while we store Docker images in our Azure Container Registry (ACR). Everytime we tag a commit in git, an amount of 22 Docker images are automatically pushed to our registry (see [this article for details](../build_using_azuredevops)). Very easy indeed, but it also causes a forever growing registry. 
While it's not a lot, it does cost money to store images in ACR: €0.003/GB/day.

We went looking for a way to save some costs on that and since we are developers, we're not going to delete images manually… It's got to be automatically!
 
We could of course follow a naive approach like deleting all images except the newest ones, but that's not what we wanted. Some people rely on older ones which is perfectly fine. 
 
We decided to keep our images in sync with the GitHub releases. Every GitHub release that is not needed anymore gets deleted, and the corresponding images are deleted from ACR.
 
## How?
These are the ingredients:

- Azure DevOps
- [NUKE](http://www.nuke.build)
- GitHub API
- Azure CLI
 
A NUKE target executes the process, it;

1.	Gets all current GitHub releases from a specific directory
    
    > Using the [GitHub Releases API](https://developer.github.com/v3/repos/releases/#list-releases-for-a-repository).

1.	Gets all current images in the corresponding ACR repositories

    > Using the [az acr repository show-tags](https://docs.microsoft.com/en-us/cli/azure/acr/repository?view=azure-cli-latest#az-acr-repository-show-tags) command

1.	Deletes all images which do not have a corresponding GitHub release (anymore)

    > Using the [az acr repository delete](https://docs.microsoft.com/en-us/cli/azure/acr/repository?view=azure-cli-latest#az-acr-repository-delete) command

The code for above steps can be found in [Build.RetentionPolicy.cs](https://github.com/avivasolutionsnl/sitecore-docker/blob/master/build/Build.RetentionPolicy.cs).

Next a Azure DevOps build pipeline triggers the NUKE target. Using the configured Azure subscription (which automatically provides us with a authentication context) we execute the NUKE `ExecuteRetentionPolicy` target.
The build pipeline can be found [here](https://github.com/avivasolutionsnl/sitecore-docker/blob/master/execute-retention-policy.yml) and is in Azure Pipelines scheduled to run every night.
 
By the way, make sure you choose the `Hosted VS2017` pool, because that's the one which has the Azure CLI already installed.
 
So that's it! No more unnecessary costs 😼.

See the live results [here](https://dev.azure.com/avivasolutions-public/sitecore-docker/_build?definitionId=3)!

## Credits
All credit for code, setup, and documentation goes to [Rob van Uden](https://www.linkedin.com/in/robvanu/) 👏.


Automatically save money by removing obsolete images from ACR 💰


Looking to automate your Sitecore Docker image build? 

We decided to leverage the free (❣), for open source software, Azure Pipelines offering from Microsoft for this. This article describes how we achieved a fully automated Sitecore Docker image build using Azure Pipelines.


Being able to build Docker images on your local machine (using [NUKE](http://www.nuke.build), see my [previous post](../build_using_nuke)) is nice, but still has some challenges. For example;

- input files need to be collected by every developer, which is error prone e.g. version differences might occur,
- with Docker for Windows the version of Docker that you use really matters, e.g. stability issues tend to come and go,
- a build consumes lots of CPU and memory, basically making your PC unusable during a build (and Sitecore XP & XC builds take around 90 minutes)
- automatically verifying that changes in a commit work

All of above challenges can be solved by having an automated build.

[Azure Pipelines](https://azure.microsoft.com/en-us/services/devops/pipelines/) offer unlimited minutes and 10 free parallel jobs for open source. As the [sitecore-docker](https://github.com/avivasolutionsnl/sitecore-docker) repo is open source and uses Microsoft Docker image, Azure Pipelines seems to be a good fit. 

Another nice feature of Azure Pipelines is that it offers configuration as code by using YAML files to specify your build. This is of great value as you can have your build fully scripted (and thus self-documenting), version controlled, and your changes reviewed in a PR.


# Build Sitecore XP
A Sitecore XP) build consists of the following steps (see in [code](https://github.com/avivasolutionsnl/sitecore-docker/blob/master/azure-build-image.yml)):

1. prepare input files
1. perform NUKE build target
1. push Docker images to a registry

## Prepare input files
A build requires the Sitecore installation files which need to be downloaded manually. The best option for this is to download these once and put them on a file share which is then used in the automated build. We chose [Azure Files](https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction) as it easy to mount (it's basically a SMB file share) on your host PC and in Azure Pipelines.

Besides the installation files, a Sitecore license file is required. We place this on the file share as well.

The Sitecore XP and XC topologies requires SSL between the services, for this we need self signed certificates for e.g. the XConnect and Solr roles. You can either let the build generate the certificates for you or generate these once and place them on the file share and have the same used for each build. See [here](https://github.com/avivasolutionsnl/sitecore-docker/blob/601f158cdbc69622b4c11ae5125ab19cdfdf4326/azure-build-image.yml#L46) how that is implemented.

## Perform NUKE build target
The nice thing about NUKE is that you can simply perform the same target on the build server as you do locally. So for example, to execute a Sitecore Xp build use the following code in the Azure Pipelines YAML file:
```
- task: PowerShell@2
    displayName: Build images
    inputs:
      targetType: 'filePath'
      filePath: build.ps1
      arguments: Xp
```
Note that it uses the `build.ps1` convenience script that NUKE provides out-of-the-box.

## Push Docker images to a registry
When you create a release you want to make the resulting Docker images externally available, i.e. in a Docker Image registry. The corresponding Azure service for this is Azure Container Registry (ACR). Once you have your ACR create in Azure, you can login and push your Docker images to it.

To login to ACR we add the following to our pipeline:
```
  - task: Docker@1
    displayName: Container registry login
    inputs:
      command: login
      azureSubscriptionEndpoint: $(azureSubscriptionEndpoint)
      azureContainerRegistry: $(azureContainerRegistry)
```

To push the images we use the same NUKE target in the Azure Pipelines build that we would do locally:
```
  - task: PowerShell@2
    displayName: (Optionally) Push images
    inputs:
      targetType: 'filePath'
      filePath: build.ps1
      arguments: PushXp --RepoImagePrefix $(dockerId)
```

NB. the push *only* performs an actual push when a Git tag exists on the current commit. This check is added by  is written in NUKE:
```
Target PushXp => _ => _
        .Requires(() => !string.IsNullOrEmpty(RepoImagePrefix))
        .OnlyWhenDynamic(() => HasGitTag() || ForcePush)
        .Executes(() => {
            PushXpImage("mssql");
            PushXpImage("sitecore");
            PushXpImage("solr");
            PushXpImage("xconnect");
        });
```

We commonly create a Github Release, which results in a Git tag, to trigger (and administer) a release build that pushes Docker images to ACR.

# Build Sitecore XP, XC, and variants
Now that we can build one Sitecore topology, i.e. XP, lets extend the build pipeline with all topologies and its variants:

- XP
- XP + SXA
- XP + JSS
- XC
- XC + SXA
- XC + JSS

This poses some challenges as a [Microsoft-hosted agent](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops) has around 10 GB of diskspace which is not enough to build all topologies. There are two options to tackle this problem:

1. Use a self-hosted agent
1. Use multiple Microsoft-hosted agents

## Self-hosted agent
A self-hosted agent is basically a VM that you configure yourself and install a Azure Devops agent on. Easiest way to get one is by using the same process as Microsoft uses, a full description of how to setup a self-hosted agent for a Sitecore Docker build can be found [here](https://github.com/avivasolutionsnl/sitecore-docker/blob/master/SELF-HOSTED-AGENT.md). 
Once you have your own agent you can scale it to your needs, e.g. adding more diskspace and CPU.
Big drawback is that you have to fully manage (e.g. setup, clean) the VM yourself.

## Multiple Microsoft-hosted agents
A nice feature of Microsoft-hosted agents is that they are pre-configured by Microsoft and automatically cleaned after a build job, something that you would have to do yourself when using a self-hosted agent. In other words, you can just use them out-of-the-box as service. Unfortunately scaling a Microsoft-hosted agent is not available at the time of writing.

We solved the diskspace problem of Microsoft-hosted agents by distributing the build over 5 agents. Each agent builds a set of images (from which some are build multiple times on different agents, but as only the resulting images are pushed this is not an issue). Using multiple build agents, we ended up with the following 5 jobs:

- XP, XC
- XP + SXA
- XC + SXA
- XP + JSS
- XC + JSS

Above distribution basically means that we have one agent for a *basic* XP/XC setup, and an additional agent for each variant (SXA, JSS).

In order to re-use a build in Azure Pipelines we use a [template](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops) named `azure-build-image.yml` in our case. This allows us to specify the same job but with different parameters, e.g:
```
jobs:
- template: azure-build-image.yml
  parameters:
    name: BaseXpXc
    buildTargets: Xc
    pushTargets: PushBase PushXp PushXc

- template: azure-build-image.yml
  parameters:
    name: XpSxa
    buildTargets: XpSxa
    pushTargets: PushXpSxa
```

The complete Azure Pipelines configuration, reusable to build your own Sitecore Docker images, can be found [here](https://github.com/avivasolutionsnl/sitecore-docker/blob/master/azure-pipelines.yml).

The live Azure Pipelines build for [sitecore-docker](https://github.com/avivasolutionsnl/sitecore-docker) can be viewed [here](https://dev.azure.com/avivasolutions-public/sitecore-docker/_build/results?buildId=115).

To summarize, to automatically build Sitecore Docker images using Azure Pipelines;

- fork https://github.com/avivasolutionsnl/sitecore-docker,
- create an Azure File share, download and add the Sitecore installation files to it,
- create an Azure Container Registry,
- create an Azure Pipeline based on the definition in the repo,

...and relax & enjoy automated builds!


November 6, 2019

XC9 + K8S + AKS = Bingo?! 🌟

May 22, 2019

Automatically save money by removing obsolete images from ACR 💰

April 15, 2019

Build Docker images using Azure Pipelines 🚀