HashiCorp Vault PKI Secrets Engine Demo for Certificate Management

This post first appeared on TeKanAid’s blog.

Overview

Certificate management is not an easy task. Most system administrators dread the day they have to work on renewing a certificate. This is because of a couple of reasons:

  • It’s risky because it may incur an outage during the process.
  • Submitting to a CA
  • Waiting for a verification and signing process to complete
Vault PKI Secrets Engine
Vault PKI Secrets Engine
Vault PKI Secrets Engine

Credit

This work is inspired by Steve Dillon’s medium blog here.

Code and Video

You can find the code for this blog post in the Vault CA demo GitHub repo. Moreover, below is a video explanation.

HashiCorp Vault PKI Secrets Engine Demo for Certificate Management

Video Chapters

You can skip to the relevant chapters below:

  • 03:43 — Terraform to create the Root and Intermediate CAs
  • 07:13 — Generate a leaf certificate for Grafana
  • 10:13 — Add the cert to Grafana
  • 11:20 — Chrome doesn’t trust the cert
  • 12:30 — Add Root & Intermediate CAs to Windows Cert Store
  • 16:30 — Chrome now trusts the Grafana cert
  • 20:30 — Revoke the cert
  • 22:20 — Clear the CRL Cache in Windows 10
  • 23:25 — Chrome shows the cert is revoked
  • 24:44 — Walkthrough of the Terraform Code
  • 36:00 — Conclusion

Pre-requisites

The following is required to follow along:

#!/bin/bash
export VAULT_ADDR=https://<url>
export VAULT_TOKEN=s.xxxxxxxxx

Create the Leaf Certificate with Vault

To create leaf certificates: Run the following command:

./create-server-certs.sh <cert_name> <common_name> <ip_sans> <TTL_in_seconds>
./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952
#!/bin/bash
# pass the name of the cert as the first argument to the script, common name as second argument, ip_sans as third, and TTL in seconds as fourth. Example: `./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952`
set -eu
echo Creating cert for $1
echo Common name: $2
echo IP Sans: $3
echo TTL: $4
source ./env-vars.shmkdir -p output/$1vault write pki-int-ca/issue/server-cert-for-home ttl=$4 common_name="$2" ip_sans="$3" -format=json > output/$1.jsoncat output/$1.json | jq -r '.data.certificate' > output/$1/$1_cert.pem
cat output/$1.json | jq -r '.data.private_key' > output/$1/$1_key.pem
cat output/$1.json | jq -r '.data.issuing_ca' > output/$1/ca.pem
cat output/$1.json | jq -r '.data.ca_chain[]' > output/$1/ca_chain.pem
# Dump the certificates in text mode
openssl x509 -noout -text -in output/$1/ca.pem > output/$1/ca.pem.txt
openssl x509 -noout -text -in output/$1/$1_cert.pem > output/$1/$1_cert.pem.txt

Moving the Certificate and the Private Key

Now that we have our certificate and key, we need to move them to the server where Grafana is running. Below is an scp command to do that.

# Example command to scp the folder containing the cert and the key from local to remote machine:
scp -r /mnt/c/Users/Sam/Deployments/HashiCorp/Vault/vault-ca-demo/output/grafana sam@192.168.1.80:/home/sam/automation/grafana/config/certs
Error Certificate Authority Invalid
Error Certificate Authority Invalid
Error Certificate Authority Invalid

Add the Root & Intermediate CAs to the Windows Certificate Store

The Windows certificate store doesn’t understand the .pem format so we need to convert to the .crt format using the script below:

#!/bin/bash
# Convert the root and int certs from .pem to .crt to be used in the Windows Certificate Store
openssl x509 -outform der -in output/root_ca/ca_cert.pem -out output/root_ca/ca_cert.crt
openssl x509 -outform der -in output/int_ca/int_cert.pem -out output/int_ca/int_cert.crt
  • Also, import the intermediate ca certificate into the Intermediate Certification Authorities -> Certificates folder
  • You may need to reboot the computer (I tried restarting the Chrome browser, but that didn’t work)
Vault Root CA added to Windows Cert Store
Vault Root CA added to Windows Cert Store
Vault Root CA added to Windows Cert Store
Vault Intermediate CA added to Windows Cert Store
Vault Intermediate CA added to Windows Cert Store
Vault Intermediate CA added to Windows Cert Store

Chrome Trusts the Grafana Certificate

Now you can once again navigate to the Grafana URL using https. Make sure you open a new incognito window to avoid caching. When you do, you’ll see that Chrome trusts the certificate issued as it trusts the entire certificate chain. The certificate chain is made up of the root CA and the intermediate CA.

Valid Grafana Certificate
Valid Grafana Certificate
Valid Grafana Certificate

Revoke the Certificate

You can revoke a certificate by following this example:

vault write pki-int-ca/revoke serial_number="62:d3:ac:77:93:25:34:11:e0:47:27:0f:d1:db:92:67:51:8c:30:3c"
vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true

Clear the CRL Cache in Windows

There is a CRL cache that needs to be flushed in Windows so that the Chrome browser reaches out to the CRL URL to check if the certificate was revoked. I found this out from this forum post.

certutil -urlcache crl
certutil -setreg chain\ChainCacheResyncFiletime @now

Chrome Shows the Certificate is Revoked

Now if you try navigating to the Grafana URL again, you won’t be able to proceed because the certificate is revoked.

Grafana Certificate is Revoked
Grafana Certificate is Revoked
Grafana Certificate is Revoked

Conclusion

In this blog post, we talked about certificate management with the Vault PKI secrets engine. We saw how easy it was to create a root and intermediate CA inside of Vault. It was also quite quick and simple to create a leaf certificate for Grafana. We didn’t need to go through the lengthy process of:

  1. Submitting to a CA
  2. Waiting for a verification and signing process to complete

References

DevSecOps and Infrastructure Automation Advocate

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store