Introducing Vaultenv: Keeping your secrets secure with Vault and Haskell

We’re pleased to announce our first bit of open source code. It is a CLI utility that fetches secrets from the HashiCorp Vault secret store. It makes secrets available using environment variables to a process of your choosing. Vaultenv generalizes fetching secrets from Vault so you don’t have to reinvent the wheel for each program in your infrastructure.

We wrote it in Haskell, and you can find it on GitHub at channable/vaultenv.

This companion post discusses:

Background

Almost every program that needs to interface with external services or databases needs API keys or access credentials. These bits of information have collectively come to be called secrets.

You likely want to control access to secrets, manage their life cycle, and audit their use. We used to store these secrets in our git repositories. This is not ideal:

These problems can be solved with a central service which manages access, life cycle, and audit trails of secrets. Such a piece of software is called a secret store.

We went with HashiCorp Vault. In Vault, secrets are encrypted at rest in a central location and available over an API. Vault is a program that you can install and run on your own infrastructure.

Some notes on terminology before we continue. A Vault secret consists of arbitrary key/value pairs, stored under a path in a storage backend. The path serves as an identifier for the secret.

We now use Vault to store API keys and access credentials as Vault secrets1.

Vault in practice

So Vault can store secrets, but how do you make these available to your programs? The default answer is to fetch them over the HTTP API. The documentation is pretty good and it works as advertised. This approach has one big architectural problem, though:

Every program that requires secrets needs to know about the Vault API.

This has a couple of consequences:

Candidate solution: generalize secret fetching and make secrets available through environment variables (like all other configuration).

Existing solutions

HashiCorp has a project, envconsul, which can fetch KV pairs from Consul and make them available through environment variables. It also supports Vault.

You can give it a list of secrets as CLI flags, and it will fetch those from Vault. In our eyes, there were a few problems. It:

We had to decide, fork and send patches - or write our own. We went with the latter. Simplicity, as well as control over the codebase and direction of the tool was a requirement here.

The tool should only do these three simple things:

Experiences

If you’re not interested in Haskell, skip straight to usage and download instructions.

We chose to write vaultenv in Haskell, because of a previous success story. It was the second project that we used Haskell for at Channable. We’d like to give another experience report.

Vaultenv was mostly written by someone on the “medium-to-advanced beginner” level. The experience was mostly positive. An advanced type system, a compiler that tells you when you have made mistakes, easy refactoring, what more could you want?

Well…

  1. Documentation of the cookbook-variety. Most libraries come with excellent API documentation. It can be difficult to get up to speed with how the author intended the functions to be used together, though. It helps to have more experienced coworkers.
  2. A Prelude that aligns with best practices. The current Prelude contains non-total functions and uses Strings for everything. It should use Maybe more often and ship faster types for working with text.

Apart from the above, there are lots of things to love about Haskell libraries. Some of my favorites:

Adding concurrency was an afterthought and a 2 line change. Want to do a bunch of HTTP requests concurrently4? Before:

newEnvOrErrors <- mapM (requestSecret opts) secrets

And after, fetching all secrets concurrently:

import Control.Concurrent.Async

newEnvOrErrors <- Async.mapConcurrently (requestSecret opts) secrets

We measured a 3x speedup because of this two line change.


Another cool thing I learned about: Lenses. It turns out you can use these without fully understanding the type theory behind them. The code is pretty readable. Think getters and setters.

Want to get the "foo" key out of the "data" dictionary in the following blob of JSON?

{
  "auth": null,
  "data": {
    "foo": "bar"
  },
  "lease_duration": 2764800,
  "lease_id": "",
  "renewable": false
}

Use a Lens!

{-# LANGUAGE OverloadedStrings #-}

import Control.Lens (preview)
import Data.Text
import qualified Data.Aeson.Lens as Lens (key, _String)
import qualified Data.ByteString.Lazy as LBS

parseResponse :: LBS.ByteString -> Maybe Text
parseResponse response =
  let
    getter = Lens.key "data" . Lens.key "foo" . Lens._String
  in
    preview getter responseBody

The OverloadedStrings extension lets us use "data" instead of pack "data" – the same goes for "foo". The extension automatically converts string literals to the right type based on the function you pass them to.


Implementing retries was a piece of cake thanks to the retry package. You can specify a RetryPolicyM, which details how often and with which delays to retry an action. Here, we use exponential backoff with jitter – a backoff pattern that causes a low amount of contention/calls5:

import qualified Control.Retry as Retry

-- We use a limited exponential backoff with the policy
-- fullJitterBackoff that comes with the Retry package.
vaultRetryPolicy :: (MonadIO m) => Retry.RetryPolicyM m
vaultRetryPolicy =
  let
    maxRetries = 9 -- Try at most 10 times in total
    baseDelayMicroSeconds = 40000
  in Retry.fullJitterBackoff baseDelayMicroSeconds
  <> Retry.limitRetries maxRetries

And then you pass this into retrying, which also expects a predicate to determine when retries should happen and the action to retry:

Retry.retrying vaultRetryPolicy shouldRetry retryAction

APIs that separate the generic from the specific are really prevalent in Haskell. As long as your action lives in MonadIO, you don’t have to change the logic of the action itself. Lovely.

How to use it

To show off Vaultenv, we need Vault itself. Download a binary from the Vault site and make sure it is in your $PATH.

Then run the following command to start a development Vault server:

$ vault server -dev

Copy the root token that has been printed to the console, we’ll need it later.

Write some secrets to the test server:

# Tell the vault client to connect over HTTP
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ vault write secret/hello foo=world bar=supersecret

Let’s try to load up the values of the foo and bar keys into a program of our choosing using vaultenv. For the purposes of this demonstration, we’ll use env – pretend it is a program you want to run to get something done.

First, we need to create a file that specifies the secrets we want vaultenv to fetch. Let’s create a file /etc/env.secrets6 with the following content:

hello#foo
BAR=hello#bar

This tells vaultenv to fetch the contents of the foo and bar keys from the hello secret. It will make each of these available through an environment variable of it’s own.

The default behaviour is to infer the name of the environment variables. The contents of the foo key will be available under HELLO_FOO. For the bar key, we tell vaultenv to use the BAR environment variable. This allows interoperability with programs that you haven’t written yourself and that expect environment variables with certain names.

We’ll invoke vaultenv as follows:

$ /usr/bin/vaultenv \
  --no-connect-tls \
  --token YOUR_VAULT_TOKEN_HERE \
  --secrets-file /etc/env.secrets \
  -- /usr/bin/env
HELLO_FOO=world
BAR=supersecret
USER=laurens
LANGUAGE=en_US
HOME=/home/laurens
[...]

Notice that:

  1. The specified secrets are available in the process environment under configurable names.
  2. Environment variables set by our shell are passed on. This means vaultenv can compose with service managers and other configuration management tools7.
  3. We need to pass --no-connect-tls to vaultenv so it can connect to the development server. It connects via HTTPS by default.
  4. The -- disambiguates between options passed to vaultenv and those passed to the program. Add flags or arguments to the program like you would expect.

In production, Vault probably runs on dedicated instances, instead of together with your program. Use the --host and --port options if you want to use something different from localhost:8200.

Conclusions and future work

Vault is a stable piece of our infrastructure at Channable. It has never stopped functioning on its own, although we had some trouble due to operator error8.

There were some gaps in tooling, so we had to write some glue code ourselves. This went pretty well. We fetch around 5.5 million secrets a day from Vault using vaultenv. Our biggest application needs around 50 secrets; fetching these generally takes between 300 and 600 milliseconds on our infrastructure.

There are some opportunities for future work:

  1. We don’t have a clear integration testing story. We currently use a bash script that sort of works, but it is not pretty. There is plenty of room for improvement here.
  2. We make two HTTP requests if we want two keys from the same secret. This rarely induces extra overhead for our own use cases – we don’t often store multiple secrets in a single path. For general use cases, this data-fetching may be optimized.

All in all, we had a nice time writing this piece of infrastructural glue. If you use HashiCorp Vault, or are thinking about using it, you might also enjoy vaultenv. Get it here.


  1. We’re not using the dynamic functionality of Vault yet, but we’re looking into that.

  2. You can, of course, create a library to implement secret fetching, but that requires that all your programs are written in the same language.

  3. This UNIX system call replaces the current process with another, while keeping the environment. This way, the tool leaves no trace and is easy to manage in systemd units, since the process is non-forking.

  4. Vault doesn’t support multi-GET, so we have to create HTTP requests for each secret. The requestSecret function has type Options -> Secret -> IO (Either VaultError EnvVar).

  5. Amazon made some observations on which retry strategies lead to the least amount of work, in terms of requests sent over time. They found that exponential backoff with jitter was a good strategy.

  6. You can name the secrets file however you like, we don’t infer anything by default. Explicit is better than implicit. Here, we have chosen the name of the program we’d like to invoke and gave it the .secrets extension.

  7. There is an open ticket to make environment inheritance configurable. Please tell us if you want this behaviour; we’ll implement it if there is demand.

  8. Secret updates are atomic, so you can’t add a field to a secret without copying all the prior fields. Not knowing this caused some secrets to be unavailable for a brief moment. Yay for backups.