Configuring Azure Blob containers as staging area

The Hyperscale Compliance Orchestrator requires a staging area to read from the source files and write to the target files. In Kubernetes deployment of Hyperscale Compliance, you can now use Azure Blob Containers as a staging area to access the Hyperscale Compliance Orchestrator and the Continuous Compliance engine’s source and target files.

This feature is supported on MicroK8s and Microsoft’s Azure Kubernetes Service (AKS) distributions and it is only available for Continuous Compliance engine version 28.0.0.0 onwards.

Prerequisites

For Hyperscale Compliance containers to access Blob objects through a file system interface, a CSI driver (Container Storage Interface) must be configured on the Kubernetes cluster.

To configure the mount point through the CSI driver, a Kubernetes cluster administrator or an Azure account administrator will need to:

Create an Azure Storage account.
Open the Azure storage account and create a Blob Container.
Create and assign Azure’s built-in roles such as Owner or Reader roles to users. For instructions, see Azure RBAC documentation.
Create a managed identity by following the instructions provided in Managed identities for Azure resources
Install the mount point for Blob CSI driver version v1.25.1 or later. For installation instructions, see Azure Kubernetes Service and MicroK8s.

Setting Up Azure Blob Containers with AKS with Managed Identity

The kubelet identity is a user-assigned managed identity that is primarily used by the AKS nodes, unless explicitly overridden, to interact with Azure resources, such as Azure Container Registry (ACR) and Azure Storage.

When creating an AKS cluster, you need to assign the kubelet identity the 'Storage Blob Data Contributor' role on the Azure Blob Container resource. This role is required for the kubelet to interact with Azure Storage for persistent volume mounting.

To enable Managed Identity on AKS cluster, Azure CLI must be installed on the environment before following the below steps. For more information, see Setup managed identity on AKS

Retrieve the Kubelet Identity

To use the Kubelet Identity for authentication, retrieve its details using the following:

CODE

az aks show --resource-group <your-resource-group> --name <your-cluster-name> --query "identityProfile.kubeletidentity"

Example output:
{
  "clientId": "<kubelet-identity-client-id>",
  "objectId": "<kubelet-identity-object-id>",
  "resourceId": "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<kubelet-identity-name>"
}

clientId: The client ID of the Kubelet Identity.
objectId: The principal ID of the Kubelet Identity.
resourceId: The resource ID of the Kubelet Identity.

Assign Roles to the Kubelet Identity

To allow the Kubelet Identity to access the Azure Blob Container, assign the Storage Blob Data Contributor role to the Kubelet Identity.

To create the respective role for the kubelet-identity, run the following:

CODE

az role assignment create \
  --assignee <kubelet-identity-object-id> \
  --role "Storage Blob Data Contributor" \
  --scope /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>

Configure the values.yaml

Provide the azureStorageIdentityClientID obtained above in values.yaml.

CODE

azureStorageIdentityClientID=<kubelet-identity-client-id>

Troubleshooting

As MicroK8s uses a different path to other standard distributions, when you are installing the Blob CSI driver on MicroK8s using:/var/snap/microk8s/common/var/lib/kubelet you may receive this error:

Could not check if "/var/snap/microk8s/common/var/lib/kubelet/pods/4a5d6dfb-646a-462d-a223-d510c059f8b7/volumes/kubernetes.io~csi/blob-pv/mount" is a mount point, Failed to read /host/proc/mounts: open /host/proc/mounts: invalid argument

If you encounter the above error, you will need to inform the /var/snap/microk8s/common/var/lib/kubelet path to driver while installing it using helm.

This is the helm chart installation command:

CODE


helm install blob-csi-driver blob-csi-driver/blob-csi-driver --set node.enableBlobfuseProxy=true -–set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" --namespace kube-system --version v1.25.1

You may also need to make the mount location shared:

CODE

/bin/mount --make-shared /var/snap/microk8s/common/var/lib/kubelet

Setup

When you have completed the steps in the prerequisites section, you will need to create a PV (persistent volume). A PV can be created via cluster administration or by using the Azure Blob related properties defined in values.yaml. Both options are discussed below:

Cluster administration creates a PV using the template below and shares the PV name which is configured against the parameter stagePvName in values.yaml.

CODE

apiVersion: v1   
kind: PersistentVolume   
metadata:   
  annotations:   
    pv.kubernetes.io/provisioned-by: blob.csi.azure.com   
  name: blob-pv   
spec:   
  capacity:   
    storage: 10Gi   
  accessModes:   
    - ReadWriteMany   
  mountOptions:   
    - -o allow_other   
    - -o direct_io   
    - --file-cache-timeout-in-seconds=0   
    - --use-attr-cache=false   
    - --invalidate-on-sync=true  
    - --attr-timeout=0 
  csi:   
    driver: blob.csi.azure.com   
    volumeHandle: azure-csi-driver-volume-handle # Should be unique for a container   
    volumeAttributes:   
      storageAccount: <STORAGE_ACCOUNT> 
      containerName: <CONTAINER_NAME> 
      AzureStorageAuthType: MSI # key, MSI   
      protocol: fuse2

Hyperscale Orchestrator creates the PV automatically during deployment using the Azure Blob related properties defined in values.yaml.

The following properties need to be specified in values.yaml even if a PV is created by cluster admin.

stagingStorageType
blobAccountName
blobContainerName
blobAuthType
blobContainerPrefix
blobContainerDelimiter
blobSecretKey
stagingAzureBlobSecretName

For more information on these parameters, see Commonly Used Properties

Additional information

Set the stagingStorageType as AZURE_BLOB_STORAGE
To connect to the Blob Container from the Continuous Compliance engine, you can use either of the following authentication mechanisms:
- Instance profile-based authentication (AZURE_MANAGED_IDENTITY)
- Access and secret key based authentication (AZURE_SECRET)
If authMechanism is set as AZURE_MANAGED_IDENTITY, a Managed Identity that grants access to the Blob Container must be attached to the Azure instances of both the Hyperscale Compliance engine, Continuous Compliance engine, or the AKS cluster. Refer to the Azure Kubernetes Services (AKS) knowledge guide for further information.
If authMechanism is set as AZURE_SECRET, you will need to provide either of the following:
- stagingAzureBlobSecretName: Name of the Kubernetes secret that holds blobSecretKey information. The stagingAzureBlobSecretName can be created by a cluster administrator using this command line:
  CODE
```
kubectl create secret generic <secret-name> --from-literal=azurestorageaccountname=<storage-account> --from-literal azurestorageaccountkey=<blob-secret--key> --type=Opaque
```
- blobSecretKey: Hyperscale Orchestrator will use these values and automatically create a generic secret.