Configuring AWS S3 bucket as staging area

The Hyperscale Compliance Orchestrator requires a staging area to read from the source file(s) and write to the target file(s). In Kubernetes deployment of Hyperscale Compliance, we can now leverage AWS S3 buckets accessible to the Hyperscale Compliance Orchestrator and the Continuous Compliance engine as the staging area. This feature is supported on microk8s and AWS EKS distributions.

Pre-requisites:

AWS provides Delphix with a Container Storage Interface (CSI) driver, and using this driver, a given Kubernetes application can access S3 objects through a file system interface. To configure the mount point through this CSI driver, a Kubernetes cluster administrator/AWS account admin will need to perform the following tasks:

1. Creation of S3 bucket

2. Creation of IAM policy

3. Creation of IAM Role

4. Installing the mount point for S3 CSI driver

 Troubleshooting tip:

If installing the S3 CSI driver on microk8s, you may encounter the following error:

CODE

Could not check if "/var/snap/microk8s/common/var/lib/kubelet/pods/4a5d6dfb-646a-462d-a223-d510c059f8b7/volumes/kubernetes.io~csi/s3-pv/mount" is a mount point, Failed to read /host/proc/mounts: open /host/proc/mounts: invalid argument

because microK8s uses /var/snap/microk8s/common/var/lib/kubelet as the kubelet path which is different from other standard distributions. We need to inform this path to driver while installing it using helm.

The helm chart installation command should be:

CODE

helm upgrade --install aws-mountpoint-s3-csi-driver --namespace kube-system aws-mountpoint-s3-csi-driver/aws-mountpoint-s3-csi-driver --set node.kubeletPath=/var/snap/microk8s/common/var/lib/kubelet

Setup:

Once the pre-requisites are met, a persistent volume needs to be created. It can be achieved in either of the following ways:

• Cluster administration creates a PV using the template below and shares the PV name which is configured against the parameter stagePvName in values.yaml.

  CODE

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: <pv-name>
  spec:
    capacity:
      storage: 5Gi # ignored, required
    accessModes:
      - ReadWriteMany
    mountOptions:
      - uid=65436
      - gid=50
      - allow-other
      - allow-delete
      - region <bucket-region>
    csi:
      driver: s3.csi.aws.com  #required
      volumeHandle: s3-csi-driver-volume
      volumeAttributes:
        bucketName: <bucket-name>

•  Hyperscale Orchestrator creates the PV automatically during deployment using the AWS S3 related properties defined in values.yaml.

The following properties shall be specified in values.yaml (even if PV is created by cluster admin).

• stagingStorageType

• authMechanism

• awsBucketName

• awsBucketRegion

• awsBucketPrefix

• awsBucketDelimiter

• awsAccessKey

• awsSecretKey

• stagingAwsS3SecretName

For more information on these parameters, see Commonly Used Properties

Notes:

• This feature is compatible only with Continuous Compliance engines v20.0.0 and above.

• The stagingStorageType must be set as AWS_S3

• The following two authentication mechanisms to connect to S3 bucket from the Continuous Compliance engine are supported:

  • Instance profile based authentication (AWS_ROLE)

  • Access and secret key based authentication (AWS_SECRET)

• If  authMechanism is set as AWS_ROLE,  an IAM instance profile, granting access to the S3 bucket, must be attached to the EC2 instances of the Hyperscale Compliance engine (or to the EKS cluster in case of EKS deployment) and the Continuous Compliance engines. For further details, refer to this AWS Knowledge Center article.

• If  authMechanism is set as AWS_SECRET, either of the following information shall be provided:

  • stagingAwsS3SecretName : Name of the Kubernetes secret that holds awsAccessKey and awsSecretKey information. It can be created by cluster admin using command line as follows:

    CODE

    kubectl create secret generic <secret-name> --from-literal=accessKeyId=<bucketAccessKeyId> --from-literal=secretAccessKey=<bucketSecretAccessKey>

  • awsAccessKey and awsSecretKey : Hyperscale Orchestrator will use these values and automatically create a generic secret.