HashiCorp Vault PKI Secrets Engine Demo for Certificate Management

Sam Gabrail

Dec 22, 2020·7 min read

This post first appeared on TeKanAid’s blog.

Overview

Certificate management is not an easy task. Most system administrators dread the day they have to work on renewing a certificate. This is because of a couple of reasons:

• It’s a manual and complex process done every few years, so it’s hard to remember everything. Therefore, you’ll find that system administrators have a method of procedure stored somewhere for when they need to go through this process again.
• It’s risky because it may incur an outage during the process.

HashiCorp Vault’s Public Key Infrastructure (PKI) secrets engine can streamline distributing TLS certificates and allows users to create PKI certificates with a single command. Vault PKI reduces the overhead around the usual manual process of:

• Generating a private key and CSR
• Submitting to a CA
• Waiting for a verification and signing process to complete

This is all done while additionally providing an authentication and authorization mechanism to validate as well.

It’s quite simple to issue certificates with Vault and we’ll show you how in this blog post. You can now create short-lived certificates, which is considered best practice from a security point of view.

We use the Vault Terraform provider to build everything. You can also opt to use the CLI, API, or Web UI to run commands to configure Vault for PKI using this guide. If you’ve been following some of my blog posts, you’ll see that I’m a big fan of using Terraform wherever possible to define our infrastructure as code.

Credit

This work is inspired by Steve Dillon’s medium blog here.

Code and Video

You can find the code for this blog post in the Vault CA demo GitHub repo. Moreover, below is a video explanation.

Video Chapters

You can skip to the relevant chapters below:

• 00:00 — Introduction
• 03:43 — Terraform to create the Root and Intermediate CAs
• 07:13 — Generate a leaf certificate for Grafana
• 10:13 — Add the cert to Grafana
• 11:20 — Chrome doesn’t trust the cert
• 12:30 — Add Root & Intermediate CAs to Windows Cert Store
• 16:30 — Chrome now trusts the Grafana cert
• 20:30 — Revoke the cert
• 22:20 — Clear the CRL Cache in Windows 10
• 23:25 — Chrome shows the cert is revoked
• 24:44 — Walkthrough of the Terraform Code
• 36:00 — Conclusion

Pre-requisites

The following is required to follow along:

• An Installation of HashiCorp Vault
• Terraform
• An Installation of Grafana on a Linux VM or in a Docker container with a Linux image

First, we need to run the env-vars.sh script to export the VAULT_ADDR and VAULT_TOKEN environment variables. We used the root token for Vault for this demo, but you should not do this in production. You would need to authenticate into Vault using your preferred authentication method.

env-vars.sh:

#!/bin/bash
export VAULT_ADDR=https://<url>
export VAULT_TOKEN=s.xxxxxxxxx

Next, we run terraform apply to create the root and intermediate CAs along with roles in Vault. For a walkthrough of the Terraform code, check this section of the video.

Create the Leaf Certificate with Vault

To create leaf certificates: Run the following command:

./create-server-certs.sh <cert_name> <common_name> <ip_sans> <TTL_in_seconds>

An example would look like this:

./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952

Below is the bash script that we ran. Notice the certificate and private key are created inside of an output folder. Make sure not to push this output folder into GitHub.

create-server-certs.sh:

#!/bin/bash
# pass the name of the cert as the first argument to the script, common name as second argument, ip_sans as third, and TTL in seconds as fourth. Example: `./create-server-certs.sh grafana docker01.home 192.168.1.80 31556952`
set -euecho Creating cert for $1
echo Common name: $2
echo IP Sans: $3
echo TTL: $4source ./env-vars.shmkdir -p output/$1vault write pki-int-ca/issue/server-cert-for-home ttl=$4 common_name="$2" ip_sans="$3" -format=json > output/$1.jsoncat output/$1.json | jq -r '.data.certificate' > output/$1/$1_cert.pem
cat output/$1.json | jq -r '.data.private_key' > output/$1/$1_key.pem
cat output/$1.json | jq -r '.data.issuing_ca' > output/$1/ca.pem
cat output/$1.json | jq -r '.data.ca_chain[]' > output/$1/ca_chain.pem# Dump the certificates in text mode
openssl x509 -noout -text -in output/$1/ca.pem > output/$1/ca.pem.txt 
openssl x509 -noout -text -in output/$1/$1_cert.pem > output/$1/$1_cert.pem.txt

Moving the Certificate and the Private Key

Now that we have our certificate and key, we need to move them to the server where Grafana is running. Below is an scp command to do that.

# Example command to scp the folder containing the cert and the key from local to remote machine:
scp -r /mnt/c/Users/Sam/Deployments/HashiCorp/Vault/vault-ca-demo/output/grafana sam@192.168.1.80:/home/sam/automation/grafana/config/certs

You then need to reload Grafana so it can pick up the certificate. Now use your browser to navigate to the Grafana site using https. You’ll notice that your browser gives you a warning message. Below is what it looks like in Chrome:

This message is basically saying that the browser doesn’t trust this certificate because it doesn’t know nor trust the CA that issued the certificate. This is because our Vault’s root and intermediate CAs are not part of the certificate store of Windows. We’ll see how to add these two CAs to the Windows certificate store in the next section.

Add the Root & Intermediate CAs to the Windows Certificate Store

The Windows certificate store doesn’t understand the .pem format so we need to convert to the .crt format using the script below:

convert-pem.sh:

#!/bin/bash
# Convert the root and int certs from .pem to .crt to be used in the Windows Certificate Store
openssl x509 -outform der -in output/root_ca/ca_cert.pem -out output/root_ca/ca_cert.crt
openssl x509 -outform der -in output/int_ca/int_cert.pem -out output/int_ca/int_cert.crt

In the Microsoft Management Console (mmc):

• You will need to import the root ca certificate into the Trusted Root Certification Authorities -> Certificates folder
• Also, import the intermediate ca certificate into the Intermediate Certification Authorities -> Certificates folder
• You may need to reboot the computer (I tried restarting the Chrome browser, but that didn’t work)

Below you can see where we’ve imported both the Vault root and intermediate CAs in the Windows certificate store.

You can also watch the video section going through the process here.

Chrome Trusts the Grafana Certificate

Now you can once again navigate to the Grafana URL using https. Make sure you open a new incognito window to avoid caching. When you do, you’ll see that Chrome trusts the certificate issued as it trusts the entire certificate chain. The certificate chain is made up of the root CA and the intermediate CA.

Revoke the Certificate

You can revoke a certificate by following this example:

vault write pki-int-ca/revoke serial_number="62:d3:ac:77:93:25:34:11:e0:47:27:0f:d1:db:92:67:51:8c:30:3c"

You can also remove a revoked certificate and clean the Certificate Revocation List (CRL) by running this command:

vault write pki_int/tidy tidy_cert_store=true tidy_revoked_certs=true

Clear the CRL Cache in Windows

There is a CRL cache that needs to be flushed in Windows so that the Chrome browser reaches out to the CRL URL to check if the certificate was revoked. I found this out from this forum post.

To view the CRL Cache:

certutil -urlcache crl

To Clear it:

certutil -setreg chain\ChainCacheResyncFiletime @now

Chrome Shows the Certificate is Revoked

Now if you try navigating to the Grafana URL again, you won’t be able to proceed because the certificate is revoked.

Conclusion

In this blog post, we talked about certificate management with the Vault PKI secrets engine. We saw how easy it was to create a root and intermediate CA inside of Vault. It was also quite quick and simple to create a leaf certificate for Grafana. We didn’t need to go through the lengthy process of:

1. Generating a private key and CSR
2. Submitting to a CA
3. Waiting for a verification and signing process to complete

We also witnessed how to revoke a certificate easily.

Most organizations have their own root CA which they use to sign an intermediate CA for Vault. They then import that intermediate CA into Vault and use Vault to issue leaf certificates. The leaf certificates issued by Vault now are trusted internally in an organization because the certificate chain is trusted based on the root CA.

You may be wondering if there is a way to automate the renewal of leaf certificates. Take a look at this post that talks about Consul-Template to Automate Certificate Management for HashiCorp Vault PKI.

References

• A step-by-step guide walkthrough for the Vault PKI secrets engine
• The GitHub repo for the code
• Consul-Template to Automate Certificate Management for Vault Blog Post

Originally published on TeKanAid’s blog on December 22, 2020.