Test and build the secrets engine

In this tutorial series, you learned how to create a new secrets engine backend, build a set of Vault roles, and create workflows to renew and revoke an API token using Vault.

Now, you will build the custom secrets engine and run it in Vault.

To do this, you will:

1. Set up your development environment.
   You will clone the HashiCups secrets engine repository. This contains many of the interfaces and objects you need to create a secrets engine.
2. Deploy the HashiCups API to Kubernetes.
   You will deploy your own version of HashiCups to Kubernetes to use the secrets engine.
3. Build the plugin.
   You will build a plugin binary for your secrets engine.
4. Register the plugin to Vault.
   You will register the plugin for your secrets engine to Vault's plugin catalog.
5. Use the secrets engine.
   You will use the secrets engine in Vault to revoke and renew JSON Web Tokens (JWTs) for the HashiCups API.

Prerequisites

• Golang 1.16+ installed and configured.
• Vault 1.8+ CLI installed locally.

If you want to run the tutorial, you need to deploy HashiCups to a live endpoint. Optional deployment requires additional infrastructure.

• A publicly accessible Kubernetes cluster.
• Kubernetes CLI installed locally.

Set up your development environment

Clone the learn-vault-plugin-secrets-hashicups repository.

$ git clone https://github.com/hashicorp-education/learn-vault-plugin-secrets-hashicups

Change into the repository directory.

$ cd vault-plugin-secrets-hashicups

Deploy the HashiCups API to Kubernetes

The Vault server needs a live API endpoint for HashiCups that does not exist locally. You can optionally deploy a live instance of HashiCups to your own Kubernetes cluster.

In your terminal, set your Kubernetes configuration to a publicly accessible cluster.

Apply the Kubernetes configuration in kubernetes/.

$ kubectl apply -f kubernetes/

You will find two applications in Kubernetes, one for product-api and the other for postgres.

$ kubectl get deployments

NAME          READY   UP-TO-DATE   AVAILABLE   AGE
postgres      1/1     1            1           91s
product-api   1/1     1            1           90s

Check that the product-api has an external IP.

$ kubectl get service product-api

NAME          TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
product-api   LoadBalancer   10.115.242.224   146.148.77.23   9090:32551/TCP   4m25s

Set the environment variable TEST_HASHICUPS_URL to the external IP of product-api.

$ export TEST_HASHICUPS_URL=http://$(kubectl get service \
  product-api -o jsonpath="{.status.loadBalancer.ingress[*].ip}"):9090

Check that you can access the HashiCups API health endpoint.

$ curl ${TEST_HASHICUPS_URL}/health

ok

Set the username you will use to test the HashiCups API as the TEST_HASHICUPS_USERNAME environment variable.

$ export TEST_HASHICUPS_USERNAME='vault-plugin-testing'

Set the password you will use to test the HashiCups API as the TEST_HASHICUPS_PASSWORD environment variable.

$ export TEST_HASHICUPS_PASSWORD='Testing!123'

Sign up for the HashiCups API with a username and password by calling the /signup API endpoint.

$ curl -X POST -H 'Content-Type:application/json' \
  ${TEST_HASHICUPS_URL}/signup \
  -d '{"username": "'${TEST_HASHICUPS_USERNAME}'", "password": "'${TEST_HASHICUPS_PASSWORD}'"}'

Output:

{"UserID":1,"Username":"vault-plugin-testing","token":"${TEST_HASHICUPS_JSON_WEB_TOKEN}"}

Build the plugin

You can build the secrets engine into a plugin and register it with a local Vault server.

Build the secrets engine into a plugin using Go.

$ go build -o vault/plugins/vault-plugin-secrets-hashicups cmd/vault-plugin-secrets-hashicups/main.go

You can find the binary in vault/plugins/.

$ ls vault/plugins/

vault-plugin-secrets-hashicups

Register the plugin to Vault

Write out the vault/server.hcl for the Vault server configuration. You need to set a plugin_directory to point to the folder with your custom secrets engine and api_addr for the plugin to communicate with Vault. The Vault instance stores everything in memory and runs locally.

$ cat > vault/server.hcl << EOF
plugin_directory = "$(pwd)/vault/plugins"
api_addr         = "http://127.0.0.1:8200"

storage "inmem" {}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = "true"
}
EOF

Start your Vault server.

$ vault server -config=vault/server.hcl

In a new terminal, set the VAULT_ADDR to the local Vault server.

$ export VAULT_ADDR='http://127.0.0.1:8200'

Initialize Vault with one key. Save the initial root token and unseal key.

$ vault operator init -key-shares=1 -key-threshold=1

Unseal Key 1: $VAULT_UNSEAL_KEY

Initial Root Token: $VAULT_TOKEN

Set the VAULT_TOKEN to the root token.

$ export VAULT_TOKEN=$VAULT_TOKEN

Unseal Vault with the unseal key.

$ vault operator unseal $VAULT_UNSEAL_KEY

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.7.3
Storage Type    inmem
Cluster Name    vault-cluster-f03f7b8d
Cluster ID      0265d7c8-30dd-daf2-501d-e0893c79b260
HA Enabled      false

Calculate the SHA256 sum of the compiled secrets engine binary. Vault uses the sum to ensure that the plugin in the catalog matches the binary in the plugin directory!

$ SHA256=$(sha256 sum vault/plugins/vault-plugin-secrets-hashicups | cut -d ' ' -f1)

$ SHA256=$(shasum -a 256 vault/plugins/vault-plugin-secrets-hashicups | cut -d ' ' -f1)

Register the plugin with Vault using the sum.

$ vault plugin register -sha256=$SHA256 secret vault-plugin-secrets-hashicups

Success! Registered plugin: vault-plugin-secrets-hashicups

Check the status of the HashiCups secrets engine by getting information from the Vault plugin catalog.

$ vault plugin info secret vault-plugin-secrets-hashicups

Key        Value
---        -----
args       []
builtin    false
command    vault-plugin-secrets-hashicups
name       vault-plugin-secrets-hashicups
sha256     $SHA256

Use the secrets engine

Set the VAULT_TOKEN environment variable to the root token.

$ export VAULT_TOKEN=$VAULT_TOKEN

Set the username you will use to test the HashiCups API as the TEST_HASHICUPS_USERNAME environment variable.

$ export TEST_HASHICUPS_USERNAME='vault-plugin-testing'

Set the password you will use to test the HashiCups API as the TEST_HASHICUPS_PASSWORD environment variable.

$ export TEST_HASHICUPS_PASSWORD='Testing!123'

Set the environment variable TEST_HASHICUPS_URL to the external IP of product-api.

$ export TEST_HASHICUPS_URL=http://$(kubectl get service \
  product-api -o jsonpath="{.status.loadBalancer.ingress[*].ip}"):9090

You can use the HashiCups secrets engine in Vault by enabling it for a path such as hashicups/.

$ vault secrets enable -path=hashicups vault-plugin-secrets-hashicups

Success! Enabled the vault-plugin-secrets-hashicups secrets engine at: hashicups/

Configure the secrets engine to access the HashiCups API.

$ vault write hashicups/config \
  username="${TEST_HASHICUPS_USERNAME}" \
  password="${TEST_HASHICUPS_PASSWORD}" \
  url="${TEST_HASHICUPS_URL}"

Output:

Success! Data written to: hashicups/config

Reading the configuration from Vault outputs the URL and username for HashiCups but not the password!

$  vault read hashicups/config

Key         Value
---         -----
url         $TEST_HASHICUPS_URL
username    $TEST_HASHICUPS_USERNAME

Create a role entry for the HashiCups secrets engine named test.

$ vault write hashicups/role/test username="${TEST_HASHICUPS_USERNAME}"

Success! Data written to: hashicups/role/test

Retrieve a new HashiCups JSON Web Token (JWT) from Vault to access the HashiCups API.

$ vault read hashicups/creds/test

Key                Value
---                -----
lease_id           hashicups/creds/test/$LEASE_ID
lease_duration     768h
lease_renewable    true
token              $HASHICUPS_JSON_WEB_TOKEN
token_id           f277c246-0c01-444c-8eb7-9ac5e2475cb7
user_id            1
username           vault-plugin-testing

Copy the value from token and save the token to the HASHICUPS_TOKEN environment variable.

$ HASHICUPS_TOKEN=$HASHICUPS_JSON_WEB_TOKEN

Add a new coffee to the HashiCups API using the token. The API successfully allows you to add a new item.

$ curl -i -X POST -H "Authorization:${HASHICUPS_TOKEN}" \
  ${TEST_HASHICUPS_URL}/coffees \
  -d '{"name":"melbourne magic", "teaser": "new to the menu", "description": "more espresso"}'

Output example:

HTTP/1.1 200 OK
Date: Wed, 18 Aug 2021 22:12:41 GMT
Content-Length: 87
Content-Type: text/plain; charset=utf-8

{"id":7,"name":"","teaser":"","description":"","price":0,"image":"","ingredients":null}

Imagine you accidentally compromised the HashiCups API token. Revoke the token from Vault.

$ vault lease revoke hashicups/creds/test/$LEASE_ID

All revocation operations queued successfully!

Try to add a new coffee to the HashiCups API using the token. The API returns a 401 Unauthorized because your secrets engine invalidated the HashiCups API token!

$ curl -i -X POST -H "Authorization:${HASHICUPS_TOKEN}" \
  ${TEST_HASHICUPS_URL}/coffees \
  -d '{"name":"burnt beans", "teaser": "overdone", "description": "good for cooking"}'

Output example:

HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Wed, 18 Aug 2021 22:15:39 GMT
Content-Length: 14

Invalid token

You successfully used your custom secrets engine to create and revoke an API token for HashiCups.

Clean up

Stop your Vault server.

Delete the Kubernetes configuration in kubernetes/.

$ kubectl delete -f kubernetes/

Next steps

Congratulations! You tested and built your own secrets engine.

If you are stuck in this tutorial, refer to the plugins/vault-plugin-secrets-hashicups/solution directory.

• To learn more about upgrading plugins, refer to the documentation on registration and reload.

• To learn more about Vault plugins, refer to the Vault Plugin System Documentation.

• Watch Developing a secrets engine for HashiCorp Vault.