Vault

Secure, store and tightly control access to tokens, passwords, certificates, and encryption keys

🚀 Introduction

HashiCorp Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, or certificates. Vault provides a unified interface to any secret, while providing tight access control and recording a detailed audit log.

In this HashiQube DevOps lab, you will get hands-on experience with HashiCorp Vault through a UI, CLI, or HTTP API.

📰 Latest News

• Vault 1.15 adopts Microsoft Workload Identity Federation
• Vault 1.14 brings ACME for PKI, AWS roles, and more improvements
• Vault 1.13 adds Kubernetes Operator, MFA improvements, and more
• Vault 1.12 Adds New Secrets Engines, ADP Updates, and More
• Vault 1.11 Adds Kubernetes Secrets Engine, PKI Updates, and More
• Vault 1.10 Achieves FIPS 140-2 Compliance

🛠️ Provision

bash vault/vault.shCopy to clipboardErrorCopied

vagrant up --provision-with basetools,docsify,vaultCopy to clipboardErrorCopied

docker compose exec hashiqube /bin/bash
bash hashiqube/basetools.sh
bash vault/vault.shCopy to clipboardErrorCopied

Once the provisioner has run, you will be able to access vault on http://localhost:8200 And you can login with the Initial Root Token displayed in the output of the provisioner.

💡 Tip: If you ever need to fetch the Initial Root Token again, you can get this inside HashiQube by doing:

• vagrant ssh
• cat /etc/vault/init.file

📊 Vault Replication Types

Vault supports two types of replication:

Performance Replication

Performance Replication allows multiple clusters to be simultaneously active in different geographical regions so applications can interact with nearby Vault servers, reducing latency when requesting secrets.

• One cluster is primary, others are secondary
• All clusters can service read requests
• Write requests are forwarded to the primary cluster
• Data flows from primary to secondary clusters

Key Concepts:

• Mount Filter: Configuration telling the primary cluster which secrets engines and auth methods should have their data replicated to specific secondary clusters
• Path Filter: Generalization of a mount filter that can filter both mounts and namespaces in Vault Enterprise
• Local Mount: A secret engine or auth method that is designated as local when created; its data is not replicated to other clusters

Disaster Recovery Replication

In Disaster Recovery Replication, only one cluster is active while the other secondary clusters serve as warm standbys in case the primary cluster suffers a catastrophic failure.

Note: Both kinds of replication can be used simultaneously.

🔒 Vault Policy Syntax

Policies define access permissions to paths in Vault. When there are potentially multiple matching policy paths, P1 and P2, the following matching criteria is applied:

1. If the first wildcard (+) or glob (*) occurs earlier in P1, P1 is lower priority
2. If P1 ends in * and P2 doesn't, P1 is lower priority
3. If P1 has more + (wildcard) segments, P1 is lower priority
4. If P1 is shorter, it is lower priority
5. If P1 is smaller lexicographically, it is lower priority

💡 Tip: The glob character is the asterisk (*). It is not a regular expression and is only supported as the last character of the path!

Capabilities

Each path must define one or more capabilities which provide fine-grained control over permitted (or denied) operations:

• create (POST/PUT) - Allows creating data at the given path
• read (GET) - Allows reading the data at the given path
• update (POST/PUT) - Allows changing the data at the given path
• patch (PATCH) - Allows partial updates to the data at a given path
• delete (DELETE) - Allows deleting the data at the given path
• list (LIST) - Allows listing values at the given path

🧩 Vault Policy Example

# Manage auth methods broadly across Vault
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Create, update, and delete auth methods
path "sys/auth/*" {
  capabilities = ["create", "update", "delete", "sudo"]
}

# List auth methods
path "sys/auth" {
  capabilities = ["read"]
}

# Create and manage ACL policies
path "sys/policies/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# To list policies - Step 3
path "sys/policies/" {
  capabilities = ["list"]
}

# List, create, update, and delete key/value secrets mounted under secret/
path "secret/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Additional policy sections are included in the original documentCopy to clipboardErrorCopied

🏢 Vault Namespaces

Vault Enterprise Namespaces allow Vault to support multi-tenant deployments in which different teams are isolated from each other and can self-manage their own secrets.

Namespaces form a hierarchy with all namespaces living under the "root" namespace. Each namespace can have its own:

• Secrets engines
• Auth methods
• Policies
• Identities
• Tokens

📊 Monitoring Vault

We use Prometheus and Grafana to Monitor Vault. See Monitoring Hashicorp Vault for details.

📚 Resources

• HashiCorp Vault Official Website

🔧 Provisioner Script

Check the complete Vault provisioner script:

# Script content available in the original file: vault.shCopy to clipboardErrorCopied

#!/bin/bash

# https://computingforgeeks.com/install-and-configure-vault-server-linux/
# https://www.vaultproject.io/

VERSION=latest

arch=$(lscpu | grep "Architecture" | awk '{print $NF}')
if [[ $arch == x86_64* ]]; then
  ARCH="amd64"
elif  [[ $arch == aarch64 ]]; then
  ARCH="arm64"
fi
echo -e '\e[38;5;198m'"CPU is $ARCH"

sudo DEBIAN_FRONTEND=noninteractive apt-get --assume-yes install -qq curl unzip jq < /dev/null > /dev/null

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Cleanup any Vault if found"
echo -e '\e[38;5;198m'"++++ "
sudo systemctl stop vault
sudo rm -rf /usr/local/bin/vault
sudo rm -rf /etc/vault
sudo rm -rf /var/lib/vault
sudo rm -rf /tmp/vault.zip

if [ -f /vagrant/vault/license.hclic ]; then
  # https://developer.hashicorp.com/vault/tutorials/enterprise/hashicorp-enterprise-license
  echo -e '\e[38;5;198m'"++++ "
  echo -e '\e[38;5;198m'"++++ Found license.hclic Installing Enterprise Edition version: $VERSION"
  echo -e '\e[38;5;198m'"++++ "
  export VAULT_LICENSE_PATH=/vagrant/vault/license.hclic
  export VAULT_LICENSE=$(cat /vagrant/vault/license.hclic)
  if [[ $VERSION == "latest" ]]; then
    LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep 'ent' | egrep "linux.*$ARCH" | sort -V | tail -n 1)
  else
    LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep 'ent' | egrep "linux.*$ARCH" | sort -V | grep $VERSION | tail -1)
  fi
else
  echo -e '\e[38;5;198m'"++++ "
  echo -e '\e[38;5;198m'"++++ Installing Community Edition version: $VERSION"
  echo -e '\e[38;5;198m'"++++ "
  if [[ $VERSION == "latest" ]]; then
    LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep -v 'rc|ent|beta' | egrep "linux.*$ARCH" | sort -V | tail -n 1)
  else
    LATEST_URL=$(curl -sL https://releases.hashicorp.com/vault/index.json | jq -r '.versions[].builds[].url' | sort -t. -k 1,1n -k 2,2n -k 3,3n -k 4,4n | egrep -v 'rc|ent|beta' | egrep "linux.*$ARCH" | sort -V | grep $VERSION | tail -1)
  fi
fi
wget -q $LATEST_URL -O /tmp/vault.zip

mkdir -p /usr/local/bin
(cd /usr/local/bin && unzip -o /tmp/vault.zip)
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Installed `/usr/local/bin/vault --version`"
echo -e '\e[38;5;198m'"++++ "

# create /var/log/vault.log
sudo touch /var/log/vault.log

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Enable Vault command autocompletion"
echo -e '\e[38;5;198m'"++++ "
vault -autocomplete-install
complete -C /usr/local/bin/vault vault

# create Vault data directories
sudo mkdir -p /etc/vault
sudo mkdir -p /var/lib/vault/data

# create user named vault
sudo useradd --system --home /etc/vault --shell /bin/false vault
sudo chown -R vault:vault /etc/vault /var/lib/vault/

# create a Vault service file at /etc/systemd/system/vault.service
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Create Vault Systemd service file"
echo -e '\e[38;5;198m'"++++ "
cat <<EOF | sudo tee /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault/config.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault/config.hcl
ExecReload=/bin/kill --signal HUP
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
LogsDirectory=vault
StandardOutput=append:/var/log/vault.log
StandardError=append:/var/log/vault.log
TimeoutStopSec=30
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Create Vault config file /etc/vault/config.hcl"
echo -e '\e[38;5;198m'"++++ "
# create Vault /etc/vault/config.hcl file
touch /etc/vault/config.hcl

# add basic configuration settings for Vault to /etc/vault/config.hcl file
# https://developer.hashicorp.com/vault/tutorials/raft/raft-storage#create-an-ha-cluster
cat <<EOF | sudo tee /etc/vault/config.hcl
ui = true
listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_disable     = true
}
# https://developer.hashicorp.com/vault/tutorials/raft/raft-storage#create-an-ha-cluster
# seal "transit" {
#   address            = "http://0.0.0.0:8200"
#   # token is read from VAULT_TOKEN env
#   # token            = ""
#   disable_renewal    = "false"
#   key_name           = "unseal_key"
#   mount_path         = "transit/"
# }
storage "raft" {
  path    = "/var/lib/vault/data"
  node_id = "hashiqube0"
}
# use a file path as storage backend
# storage "file" {
#   path  = "/var/lib/vault/data"
# }
# use consul as storage backend
# storage "consul" {
#  address = "127.0.0.1:8500"
#  path    = "vault"
# }
# https://developer.hashicorp.com/vault/docs/configuration/telemetry
# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
telemetry {
  disable_hostname = true
  prometheus_retention_time = "12h"
}
api_addr             = "http://10.9.99.10:8200"
max_lease_ttl        = "10h"
default_lease_ttl    = "10h"
cluster_name         = "hashiqube"
cluster_addr         = "http://10.9.99.10:8201"
raw_storage_endpoint = true
disable_cache        = true
disable_mlock        = true
disable_sealwrap     = true
disable_printable_check = true
EOF
if [ -f /vagrant/vault/license.hclic ]; then
  echo "license_path = \"/vagrant/vault/license.hclic\"" >> /etc/vault/config.hcl
fi

# start and enable vault service to start on system boot
sudo systemctl daemon-reload
sudo systemctl enable --now vault
sleep 20

# check vault status
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Check Vault Status"
echo -e '\e[38;5;198m'"++++ "
sudo systemctl status vault | cat

# initialize vault server
export VAULT_ADDR=http://127.0.0.1:8200
echo "export VAULT_ADDR=http://127.0.0.1:8200" >> ~/.bashrc

# start initialization with the default options by running the command below
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Initialize Vault"
echo -e '\e[38;5;198m'"++++ "
sudo rm -rf /etc/vault/init.file
vault operator init > /etc/vault/init.file

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Auto unseal vault"
echo -e '\e[38;5;198m'"++++ "
for i in $(cat /etc/vault/init.file | grep Unseal | cut -d " " -f4 | head -n 3); do vault operator unseal $i; done
vault status
cat /etc/vault/init.file
# add vault ENV variables
VAULT_TOKEN=$(grep 'Initial Root Token' /etc/vault/init.file | cut -d ':' -f2 | tr -d ' ')
grep -q "${VAULT_TOKEN}" /etc/environment
if [ $? -eq 1 ]; then
  echo "VAULT_TOKEN=${VAULT_TOKEN}" >> /etc/environment
else
  sed -i "s/VAULT_TOKEN=.*/VAULT_TOKEN=${VAULT_TOKEN}/g" /etc/environment
fi
grep -q "VAULT_ADDR=http://127.0.0.1:8200" /etc/environment
if [ $? -eq 1 ]; then
  echo "VAULT_ADDR=http://127.0.0.1:8200" >> /etc/environment
else
  sed -i "s%VAULT_ADDR=.*%VAULT_ADDR=http://127.0.0.1:8200%g" /etc/environment
fi

export VAULT_TOKEN=${VAULT_TOKEN}

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Vault status"
echo -e '\e[38;5;198m'"++++ "
vault status

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Cat Vault Credentials"
echo -e '\e[38;5;198m'"++++ "
cat /etc/vault/init.file
if [ -f /vagrant/vault/license.hclic ]; then
  echo -e '\e[38;5;198m'"++++ "
  echo -e '\e[38;5;198m'"++++ Vault License Inspect"
  echo -e '\e[38;5;198m'"++++ "
  vault license inspect /vagrant/vault/license.hclic
fi

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Access Vault"
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Vault Initial Root Token: ${VAULT_TOKEN}"
echo -e '\e[38;5;198m'"++++ Vault http://localhost:8200/ui and enter the Root Token displayed above"
echo -e '\e[38;5;198m'"++++ Vault Documentation http://localhost:3333/#/vault/README?id=vault"

# TODO: FIXME
# https://www.vaultproject.io/docs/secrets/ssh/signed-ssh-certificates
echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Lets use Vault for Signed SSH Certificates"
echo -e '\e[38;5;198m'"++++ "
echo -e "\n"

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ vault secrets enable -path=ssh-client-signer ssh"
echo -e '\e[38;5;198m'"++++ "
vault secrets enable -path=ssh-client-signer ssh

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ vault write ssh-client-signer/config/ca generate_signing_key=true"
echo -e '\e[38;5;198m'"++++ "
vault write ssh-client-signer/config/ca generate_signing_key=true

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem"
echo -e '\e[38;5;198m'"++++ "
vault read -field=public_key ssh-client-signer/config/ca | sudo tee /etc/ssh/trusted-user-ca-keys.pem

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Add TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem to /etc/ssh/sshd_config and reload SSH"
echo -e '\e[38;5;198m'"++++ "
grep -q "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" /etc/ssh/sshd_config
if [ $? -eq 1 ]; then
  echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" | sudo tee -a /etc/ssh/sshd_config
else
  sudo sed -i "s/TrustedUserCAKeys \/etc\/ssh\/trusted-user-ca-keys.pe/TrustedUserCAKeys \/etc\/ssh\/trusted-user-ca-keys.pe/g" /etc/ssh/sshd_config
fi

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Restart SSH"
echo -e '\e[38;5;198m'"++++ "
sudo systemctl reload ssh

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Create a named Vault role for signing client keys"
echo -e '\e[38;5;198m'"++++ "
vault write ssh-client-signer/roles/my-role -<<EOH
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "30m0s"
}
EOH

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Generate the SSH public key for user ubuntu"
echo -e '\e[38;5;198m'"++++ "
sudo -H -u ubuntu ssh-keygen -q -t rsa -N '' <<< ""$'\n'"y" 2>&1 >/dev/null

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ Ask Vault to sign this created public key"
echo -e '\e[38;5;198m'"++++ "

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ vault write ssh-client-signer/sign/my-role public_key=@/home/ubuntu/.ssh/id_rsa.pub"
echo -e '\e[38;5;198m'"++++ "
sudo -H -u ubuntu vault write ssh-client-signer/sign/my-role public_key=@/home/ubuntu/.ssh/id_rsa.pub
sudo -H -u ubuntu vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@/home/ubuntu/.ssh/id_rsa.pub | sudo -H -u ubuntu tee /home/ubuntu/.ssh/id_rsa-cert.pub

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ View enabled extensions, principals, and metadata of the signed key"
echo -e '\e[38;5;198m'"++++ ssh-keygen -Lf /home/ubuntu/~/.ssh/id_rsa-cert.pub"
echo -e '\e[38;5;198m'"++++ "
sudo -H -u ubuntu ssh-keygen -Lf /home/ubuntu/.ssh/id_rsa-cert.pub

echo -e '\e[38;5;198m'"++++ "
echo -e '\e[38;5;198m'"++++ SSH to localhost using the signed key"
echo -e '\e[38;5;198m'"++++ sudo -H -u ubuntu ssh -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ubuntu/.ssh/id_rsa-cert.pub -i /home/ubuntu/.ssh/id_rsa ubuntu@localhost"
echo -e '\e[38;5;198m'"++++ "
sudo -H -u ubuntu ssh -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /home/ubuntu/.ssh/id_rsa-cert.pub -i /home/ubuntu/.ssh/id_rsa ubuntu@localhost || true
echo $?

# https://www.vaultproject.io/docs/secrets/ssh/dynamic-ssh-keys
#sudo apt-get -y install pwgen
#sudo useradd -m -p $(openssl passwd -1 $(pwgen)) -s /bin/bash ubuntu
#echo "ubuntu ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ubuntu
#vault secrets enable ssh
#sudo -H -u ubuntu vault write ssh/keys/vault_key key=@/home/ubuntu/.ssh/id_rsa
#vault write ssh/roles/dynamic_key_role key_type=dynamic key=vault_key admin_user=ubuntu default_user=ubuntu cidr_list=0.0.0.0/0

#echo -e '\e[38;5;198m'"++++ Please run the following on your local computer" 
#echo -e '\e[38;5;198m'"++++ export VAULT_TOKEN=$(grep 'Initial Root Token' /etc/vault/init.file | cut -d ':' -f2 | tr -d ' ')"
#echo -e '\e[38;5;198m'"++++ export VAULT_ADDR=http://10.9.99.10:8200"
#echo -e '\e[38;5;198m'"++++ vagrant ssh -c \"vault write ssh/creds/dynamic_key_role ip=10.9.99.10\""

# check vault status
# vault status

# replace “s.BOKlKvEAxyn5OS0LvfhzvBur” with your Initial Root Token stored in the /etc/vault/init.file file
# export VAULT_TOKEN="s.RcW0LuNIyCoTLWxrDPtUDkCw"

# enable approle authentication
# vault auth enable approle
# Success! Enabled approle auth method at: approle/

# same command can be used for other Authentication methods, e.g

# vault auth enable kubernetes
# Success! Enabled kubernetes auth method at: kubernetes/

# vault auth enable userpass
# Success! Enabled userpass auth method at: userpass/

# vault auth enable ldap
# Success! Enabled ldap auth method at: ldap/

# list all Authentication methods using the command
# vault auth list

# get secret engine path:
# vault secrets list

# write a secret to your kv secret engine.
# vault kv put secret/databases/db1 username=DBAdmin
# Success! Data written to: secret/databases/db1

# vault kv put secret/databases/db1 password=StrongPassword
# Success! Data written to: secret/databases/db1

# you can even use single line command to write multiple data.
# vault kv put secret/databases/db1 username=DBAdmin password=StrongPassword
# Success! Data written to: secret/databases/db1

# to get a secret, use vault get command.
# vault kv get secret/databases/db1

# get data in json format:
# vault kv get -format=json secret/databases/db1

# to print only the value of a given field, use:
# vault kv get -field=username  secret/databases/db1

# to delete a Secret, use:
# vault kv delete secret/databases/db1
# Success! Data deleted (if it existed) at: secret/databases/db1

# vault kv get secret/databases/db1
# No value found at secret/databases/db1
Copy to clipboardErrorCopied