The Engineering Blog

Keeping Secrets in Plain Sight

by Nejc Saje · 09 Aug 2016

What is a secret? It’s information we don’t want others to know. In the context of server side applications the secrets are database passwords, API tokens, private keys and other information that we want to keep secret.

Many of such secrets are specified in the configuration files, alongside items that don’t necessarily need to be secret, such as database names, debug flags and other configuration.

As a result, production configuration files are often kept encrypted in a separate repository then decrypted at deploy time. This approach works, but becomes cumbersome when the non-secret configuration is frequently changed or when new secrets are often added to the application. Because every change to the configuration involves re-encrypting the file, we need to deal with both a separate workflow for these changes and losing the ability to git blame individual configuration properties.

At Zemanta, we wanted a way of managing secrets that would enable us to:

  • add a new secret to an application in a straightforward and simple manner,
  • enable quick and simple configuration changes,
  • update a secret and still be able to inspect previous versions,
  • attribute a secret to an application,
  • use one pattern to manage secrets across all of our applications.

SecretCrypt

To address the above points, we created SecretCrypt, a tool for keeping encrypted secrets inside configuration files that reside in the app’s codebase (and by extension in its code versioning system of choice like Git). The secrets are then decrypted on the fly when the application is started.

For example, a Django settings.py file containing secrets looks something like this:

from secretcrypt import Secret

# the encrypted secret
GOOGLE_OAUTH_CLIENT_SECRET = Secret('kms:region=us-east-1:CiDdfiP9DHF5...rQG/YJrGQ==').get()

# other, non-secret configuration
GOOGLE_OAUTH_ENABLED = True
GOOGLE_OAUTH_CLIENT_ID = '22398899238-123ji2o1i4u2198042jio.apps.googleusercontent.com'

Since the configuration file is kept in the same repository as the code, configuration options or secrets can easily be changed or added by developers themselves. We also get the history of all the changes as an extra feature, since secrets are subjected to version control and the encryption is performed per-secret, not per-file.

How it works

To achieve this, we make use of AWS’s Key Management Service (KMS). KMS allows us to create encryption keys that can then be used with the service’s encrypt and decrypt endpoints. Permissions to encrypt or decrypt something with a given key are mandated by IAM policies. In our case, we’ve set up key permissions in such a way that our developers have the permission to encrypt new secrets, but cannot decrypt them. On the other hand, the EC2 instances that run the application are assigned an IAM role that has the permission to decrypt those secrets.

Alternative secret backends

SecretCrypt is designed in a modular fashion, so it can support multiple encryption/decryption backends. Alongside KMS it currently also supports local encryption, which uses locally generated keys for AES encryption and is intended for local development purposes.

An alternative to the KMS backend could also be HashiCorp’s Vault, which is a self-hosted encryption-as-a-service (among other things).

Adding a new secret

A new secret can be encrypted using a CLI tool by simply passing the alias of our KMS encryption key:

$ encrypt-secret kms alias/Z1secrets
Enter plaintext: VerySecretValue!
kms:region=us-east-1:CiC/SXeuXDGR...

and then included in the Python file as

from secretcrypt import Secret

MY_SECRET = Secret('kms:region=us-east-1:CiC/SXeuXDGR...').get()

When the Python file is loaded, SecretCrypt will use KMS to decrypt the secret. It uses boto, so it supports AWS credentials from various sources, such as local configuration, environment variables, EC2 roles, etc. (view more)

Go version

Since we wanted to use a single secrets workflow for all our applications, we also developed SecretCrypt for Go. It implements the TextUnmarshaler interface, so it supports various configuration file formats such as TOML,

MySecret = "kms:region=us-east-1:CiC/SXeuXDGRADRIjc0qcE...

YAML,

mysecret: kms:region=us-east-1:CiC/SXeuXDGRADRIjc0qcE...

or JSON.

{"MySecret": "kms:region=us-east-1:CiC/SXeuXDGRADRIjc0qcE..."}

The secret is then used in the application’s config struct:

type Config struct {
    MySecret secretcrypt.Secret
}

var conf Config
if _, err := toml.Decode(tomlData, &conf); err != nil {
    // handle error
}
plaintext := conf.MySecret.Get()

Conclusion

In the end, developer tooling is all about reducing friction. SecretCrypt has allowed our developers to tweak configuration in production without having to re-encrypt the secrets. It has also enabled them to add new secrets without the assistance of the ops team. We are now able to track the history of the changes of both our configuration and our secrets.

We have been using SecretCrypt in production for almost six months now! We were able to drop a separate encrypted Git repository and the corresponding workflow that we used for configuration previously. Simply change, commit, and deploy!

Discuss on Hacker News

Related Posts