Using HashiCorp Vault for storing secrets

HashiCorp Vault is an open-source tool for securely storing and accessing secrets (e.g., passwords, certificates, and tokens).

Configure storage of secrets and access to them in a Yandex Managed Service for Kubernetes cluster using a Yandex Cloud Marketplace product called HashiCorp Vault with Key Management Service support.

This guide describes an example of mounting a secret from HashiCorp Vault using a Container Storage Interface (CSI) volume.

To enable access to a secret in a Managed Service for Kubernetes cluster using HashiCorp Vault:

1. Prepare your cloud.
2. Install HashiCorp Vault.
3. Log in to HashiCorp Vault.
4. Create a secret.
5. Configure the Kubernetes authentication method.
6. Install the SCI driver for a secret storage.
7. Create the SecretProviderClass resource.
8. Create a pod with a mounted secret.

If you no longer need the resources you created, delete them.

Prepare your cloudPrepare your cloud

1. Create a Kubernetes cluster and node group.

   Manually

   Using Terraform

   1. If you do not have a network yet, create one.

   2. If you do not have any subnets yet, create them in the availability zones where your Kubernetes cluster and node group will be created.

   3. Create service accounts:

      • Service account with the k8s.clusters.agent and vpc.publicAdmin roles for the folder where the Kubernetes cluster is created. This service account will be used to create the resources required for the Kubernetes cluster.
      • Service account with the container-registry.images.puller role. Nodes will pull the required Docker images from the registry on behalf of this account.

      Tip

      You can use the same service account to manage your Kubernetes cluster and its node groups.

   4. Create security groups for the Managed Service for Kubernetes cluster and its node groups.

      Warning

      The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

   5. Create a Kubernetes cluster and a node group in any suitable configuration. When creating it, specify the security groups prepared earlier.

   1. If you do not have Terraform yet, install it.

   2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

   3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

   4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

   5. Download the k8s-cluster.tf cluster configuration file to the same working directory. The file describes:

      • Network.

      • Subnet.

      • Kubernetes cluster.

      • Service account required to use the Managed Service for Kubernetes cluster and node group.

      • Security groups which contain rules required for the Managed Service for Kubernetes cluster and its node groups.

        Warning

        The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

   6. In k8s-cluster.tf, specify:

      • Folder ID.
      • Kubernetes version for the Kubernetes cluster and node groups.
      • Kubernetes cluster CIDR.
      • Name of the Managed Service for Kubernetes cluster service account.

   7. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   8. Create the required infrastructure:

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

2. Install kubectl and configure it to work with the created cluster.

Install HashiCorp VaultInstall HashiCorp Vault

Install HashiCorp Vault using Helm and initialize the vault according to the guide. In the installation command, specify the hcv namespace and add parameters to activate the Vault CSI provider mechanism:

--namespace hcv \
--set "injector.enabled=false" \
--set "csi.enabled=true"

Log in to HashiCorp VaultLog in to HashiCorp Vault

1. Run a session of the HashiCorp Vault interactive shell for the hashicorp-vault-0 pod.

   kubectl exec -it hashicorp-vault-0 \
      --namespace hcv \
      -- /bin/sh
   

2. Print (unseal) the storage.

   vault operator unseal
   

   Enter one of the recovery keys (Recovery Key) obtained during storage initialization.

3. Log in to HashiCorp Vault using the root token:

   vault login
   

   Enter the root token (Initial Root Token) obtained during storage initialization.

Create a secretCreate a secret

1. Enable the kv secret mechanism at the secret path:

   vault secrets enable -path=secret kv
   

2. Create a secret at the secret/db-pass path. Specify a password as a secret:

   vault kv put secret/db-pass password="12345678"
   

3. Make sure that the secret is available for reading at the secret/db-pass path:

   vault kv get secret/db-pass
   

   Result:

   ====== Data ======
   Key         Value
   ---         -----
   password    12345678
   

Configure the Kubernetes authentication methodConfigure the Kubernetes authentication method

This method will allow you to log in using a token of a Kubernetes service account.

1. Enable the Kubernetes authentication method:

   vault auth enable kubernetes
   

2. Configure authentication with Kubernetes API address:

   vault write auth/kubernetes/config \
      kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
   

   The KUBERNETES_PORT_443_TCP_ADDR environment variable refers to the internal network address of the Kubernetes node.

3. Create the internal-app policy that will allow the Kubernetes service account to read the previously created secret:

   vault policy write internal-app - <<EOF
   path "secret/db-pass" {
     capabilities = ["read"]
   }
   EOF
   

4. Create the database role that will link the internal-app policy to the webapp-sa Kubernetes service account (you will create it later):

   vault write auth/kubernetes/role/database \
      bound_service_account_names=webapp-sa \
      bound_service_account_namespaces=hcv \
      policies=internal-app \
      ttl=20m
   

   Tokens returned after authentication will be valid for 20 minutes.

5. Exit HashiCorp Vault:

   exit
   

Install the SCI driver for a secret storageInstall the SCI driver for a secret storage

1. Add a Helm repository named secrets-store-csi-driver:

   helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

2. Install the SCI driver:

   helm install csi secrets-store-csi-driver/secrets-store-csi-driver \
      --namespace=hcv \
      --set syncSecret.enabled=true
   

3. Make sure that the driver is running and ready:

   kubectl get pods -n hcv -l "app=secrets-store-csi-driver"
   

   Result:

   NAME                                 READY   STATUS    RESTARTS   AGE
   csi-secrets-store-csi-driver-nbxcd   3/3     Running   0          4m28s
   

Create the SecretProviderClass resourceCreate the SecretProviderClass resource

1. Create a file named spc-vault-database.yaml with settings to be transmitted to the CSI provider:

   spc-vault-database.yaml

   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: vault-database
   spec:
     provider: vault
     parameters:
       vaultAddress: "http://hashicorp-vault.hcv:8200"
       roleName: "database"
       objects: |
         - objectName: "db-password"
           secretPath: "secret/db-pass"
           secretKey: "password"
   

2. Create the SecretProviderClass resource:

   kubectl apply -f spc-vault-database.yaml -n hcv
   

Create a pod with a mounted secretCreate a pod with a mounted secret

1. Create a service account named webapp-sa in the Kubernetes cluster:

   kubectl create serviceaccount webapp-sa \
      --namespace hcv
   

2. Create a file named webapp-pod.yaml containing the webapp pod configuration:

   spc-vault-database.yaml

   kind: Pod
   apiVersion: v1
   metadata:
     name: webapp
   spec:
     serviceAccountName: webapp-sa
     containers:
     - image: jweissig/app:0.0.1
       name: webapp
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
     volumes:
       - name: secrets-store-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "vault-database"
   

3. Create the webapp pod:

   kubectl apply -f webapp-pod.yaml -n hcv
   

4. Make sure that the webapp pod is running and ready:

   kubectl get pod webapp -n hcv
   

   Result:

   NAME     READY   STATUS    RESTARTS   AGE
   webapp   1/1     Running   0          5m25s
   

5. Display the secret password recorded in the file system at the /mnt/secrets-store/db-password path:

   kubectl exec webapp -n hcv -- cat /mnt/secrets-store/db-password
   

   Result:

   12345678
   

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them.

See alsoSee also

• HashiCorp Vault documentation
• Installing HashiCorp Vault with Key Management Service support
• Installing the External Secrets Operator with Yandex Lockbox support
• Syncing with Yandex Lockbox secrets