Categories
DevOps secrets

Storing Secrets for Teams

tl;dr: Use Hashicorp Vault

Storing secrets among teams is important. There is a great blog on managing secrets and the things you shouldn’t do which I will highlight here:

  • Using the same password for all the things!
  • Using a shared excel file of secrets
  • Emailing passwords around
  • Using Chat to store secrets
  • Using Git repos to store secrets
  • Custom secrets management built inhouse

You can view the options the author gives about different ways to store secrets

Measuring the Stored Secret Tools

An important way of choosing which tool to use for your team is actually using it. You will get a feel for the experience and whether it feels robust. I also think there should be a list of predefined objective measures or requirements from the tool. Combining the tools that meet the requirements and that also feel good will guide our choice.

In our case we have the following requirements and measures:

  • Support LDAP
  • Self-hosted, open source and not too expensive
  • Level of good security practice
  • Audit trail
  • Role organsation and management
  • Types of secrets it can store
  • User Experience
  • API existance, level of API documentation and integration
  • documentation of system
  • Nice to haves: CLI tool, one time password link

The Tools

Tools I found for stored secrets but with much more (cmdb):

Tools I found specifically for stored secrets for teams:

Addons to tools I found:

LDAP/SSH tools:

Netbox

An excellent CMDB (Configuration Management Database) and Ipam (IP Address Management) that does all the things.

It was hard to use though, I was not able to create a secret as you needed a device first and setting up a device didn’t work.

This solution may work, it just didn’t in my case.

Vault

  • Support LDAP: Yes
  • Self-hosted, open source and not too expensive: Self hosted, open source and there is a paid version.
  • Level of good security practice: Hard to beat
  • Audit trail: Yes but limited to file, syslog and socket.
  • Role organsation and management: Fine grained with policies and group mappings linked to LDAP groups
  • Types of secrets it can store: Has secret engines with defaults like: Key-value, PKI certificates, SSH,customer secrets, dynamic secrets for db’s
  • User Experience: Decent, clean
  • API existance, level of API documentation and integration: Good albeit complicated
  • documentation of system: Excellent albeit a bit complicated
  • Nice to haves:
    • dynamic secrets – time-limited (leased) access to other systems
    • multiple authentication methods – can have multiple LDAP auth configs
    • Process of locking down or sealing a vault – good practice

vault-available-secret-engines

KeyWhiz

A system for managing and distributing secrets. Built with java.

I tried with the docker and maven ways and no success.  Just errors:


[ERROR] Failed to execute goal org.codehaus.mojo:sql-maven-plugin:1.5:execute (default) on project keywhiz-model: Access denied for user 'root'@'localhost' (using password: NO) -> [Help 1]

sysPass

Systems Password Manager. Built in PHP.

I used docker, it is important during setup to set server/host to: syspass-db

With the docker setup I enabled ldap and after auth, it goes stright to http://localhost:32770/undefined with a 404. Something to do with this open issue.

I’m also not a fan of material design which is already old and has never felt clean to me.

KeyCloak

Keycloak is not a password manager. It is open source identity and access management – allowing for single sign on.

Features include:

  • User Registration
  • Social Login
  • Single sign on for all applications in the same realm
  • 2 factor auth
  • LDAP integration
  • Kerberos broker
  • multitenancy and per realm customisable skin

I used the docker container for this appliation and everything just worked and it was clean.

It feels like it is the gold standard in IAM and single sign on.

I’ve haven’t used IAM or SSO yet, but this is a high quality product/project just based on the fact that the docker container just worked.

BitWarden

Open source password management solutions

To use it for organisations/teams you need the enterprise edition. I didn’t try this one.

Goldfish frontend for Vault

A HashiCorp Vault UI

I’m not sure why we need this, seems everything you can do with this can be done with the standard vault web ui.

Psono

Password Manager. Psono client is JS, server is python.

  • Support LDAP: No, unless you pay.
  • Self-hosted, open source and not too expensive: Self hosted. Opensource, start paying $2 per user per month after 10 users.
  • Level of good security practice: ?
  • Audit trail: No, unless you pay.
  • Role organsation and management: Decent
  • Types of secrets it can store: Password, Note, GPG Key, Bookmark, File (Couldn’t
  • User Experience: Looks good, not too complicated
  • API existance, level of API documentation and integration: ?
  • documentation of system: Has documentation, not sphinx docs. It’s own shitty docs.
  • Nice to haves:
    • Callbacks – hits a url when a password is changed

Overall it is missing the mark a bit…

Teampass

A PHP based collaborative passwords manager

  • Support LDAP: Yes
  • Self-hosted, open source and not too expensive: Yes, open source. No enterprise version.
  • Level of good security practice: ?
  • Audit trail: Decent – Can view connections, passwords access and by who
  • Role organsation and management: Fine grained – but not linked to LDAP groups
  • Types of secrets it can store: Restricted to passwords
  • User Experience: Decent, a bit clunky – it is php
  • API existance, level of API documentation and integration: Yes, but not great
  • Documentation of system: Good, sphinx readthedocs
  • Nice to haves:
    • Password expiration
    • Import/Export
    • One time view password link
    • Web browser extension
    • Automatic logout

Passbolt

Couldn’t get the docker install to work…

Lyft Confidant

Your secret keeper. Written in python and JS.

  • Support LDAP: No. But has KMS, google and SAML – whatever that means.
  • Self-hosted, open source and not too expensive: Self hosted, open source. No enterprise edition.
  • Level of good security practice: ?
  • Audit trail: Yes, looks pretty basic though
  • Role organsation and management: ?
  • Types of secrets it can store: Seems you can store anything
  • User Experience: Ok – not very intuitive or guiding
  • API existance, level of API documentation and integration: API exists, it is badly documented
  • documentation of system: Home made docs, basics skipped.
  • Nice to haves: CLI tool, one time password link
    • Works closely with AWS IAM
    • Blind credentials

 

ssh-ldap-pubkey

This is a python command line utility that can get ssh keys from your ldap server and add them to your server’s authorized keys.

Documentation on ssh-ldap-pubkey

This tool allowed me to view the SSH public keys for people on ldap.

It was however not automatic…I had to install the package globally. Then I had to manually configure /etc/ldap.conf with the ldap server details.

On a fresh ubuntu you might also have to install:

sudo apt-get install libsasl2-dev python-dev libldap-dev libssl-dev

Another annoying thing is I had to install a specific version 1.3.0 as 1.3.1 was borked.
Then to configure OpenSSH server to fetch users’ authorized keys  you had to add 2 lines to /etc/ssh/sshd_config:

AuthorizedKeysCommand /usr/bin/ssh-ldap-pubkey-wrapper
AuthorizedKeysCommandUser nobody

Still didn’t manage to get it to work

What happens in these cases:

  1. A new user tries to ssh to your server with their ldap username?
  2. A user whose ssh key has been added to your server is removed or taken out of the group on ldap?

Conclustion

Use Hashicorp Vault – start small and grow out your secrets infrastructure eventually branching out into dynamic secrets and public key infrastructure.

Extras

A video you can watch on the topic: