Author: stephen

Storing Secrets for Teams

Storing secrets among teams is important. There is a great blog on managing secrets and the things you shouldn’t do which I will highlight here:

  • Using the same password for all the things!
  • Using a shared excel file
  • Emailing passwords around
  • Using Chat
  • Using Git repos
  • Custom secrets management built in house

You can view the options the author gives about different ways to store secrets

Measuring the Stored Secret Tools

An important way of choosing which tool to use for your team is actually using it. You will get a feel for the experience and whether it feels robust. I also think there should be a list of predefined objective measures or requirements from the tool. Combining the tools that meet the requirements and that also feel good will guide our choice.

In our case we have the following requirements and measures:

  • Support LDAP
  • Self-hosted, open source and not too expensive
  • Level of good security practice
  • Audit trail
  • Role organsation and management
  • Types of secrets it can store
  • User Experience
  • API existance, level of API documentation and integration
  • documentation of system
  • Nice to haves: CLI tool, one time password link

The Tools

Tools I found for stored secrets but with much more (cmdb):

Tools I found specifically for stored secrets for teams:

Addons to tools I found:

LDAP/SSH tools:




  • Support LDAP: Yes
  • Self-hosted, open source and not too expensive:
  • Level of good security practice
  • Audit trail
  • Role organsation and management
  • Types of secrets it can store
  • User Experience
  • API existance, level of API documentation and integration
  • documentation of system
  • Nice to haves:
    • dynamic secrets – time-limited access to other systems
    • multiple authentication methods – can have multiple LDAP auth configs


A system for managing and distributing secrets. Built with java.

I tried with the docker and maven ways and no success.  Just errors:

[ERROR] Failed to execute goal org.codehaus.mojo:sql-maven-plugin:1.5:execute (default) on project keywhiz-model: Access denied for user 'root'@'localhost' (using password: NO) -> [Help 1]




Systems Password Manager. Built in PHP.

I used docker, importatn during setup to set server/host to: syspass-db

With the docker setup I enabled ldap and after auth, it goes stright to http://localhost:32770/undefined with a 404. Something to do with this open issue.

I’m also not a fan of material design which is already old and has never felt clean to me.


Keycloak is not a password manager. It is open source identity and access management – allowing for single sign on.

Features include:

  • User Registration
  • Social Login
  • Single sign on for all applications in the same realm
  • 2 factor auth
  • LDAP integration
  • Kerberos broker
  • multitenancy and per realm customisable skin

I used the docker container for this appliation and everything just worked and it was clean.

It feels like it is the gold standard in IAM and single sign on.

I’ve haven’t used IAM or SSO yet, but this is a high quality product/project just based on the fact that the docker container just worked.


Open source password management solutions

To use it for organisations/teams you need the enterprise edition. I didn’t try this one.


Password Manager. Psono client is JS, server is python.

  • Support LDAP: No, unless you pay.
  • Self-hosted, open source and not too expensive: Self hosted. Opensource, start paying $2 per user per month after 10 users.
  • Level of good security practice: ?
  • Audit trail: No, unless you pay.
  • Role organsation and management: Decent
  • Types of secrets it can store: Password, Note, GPG Key, Bookmark, File (Couldn’t
  • User Experience: Looks good, not too complicated
  • API existance, level of API documentation and integration: ?
  • documentation of system: Has documentation, not sphinx docs. It’s own shitty docs.
  • Nice to haves:
    • Callbacks – hits a url when a password is changed

Overall it is missing the mark a bit…


A PHP based collaborative passwords manager

  • Support LDAP: Yes
  • Self-hosted, open source and not too expensive: Yes, open source. No enterprise version.
  • Level of good security practice: ?
  • Audit trail: Decent – Can view connections, passwords access and by who
  • Role organsation and management: Fine grained
  • Types of secrets it can store: Restricted to passwords
  • User Experience: Decent, a bit clunky – it is php
  • API existance, level of API documentation and integration: Yes, but not great
  • Documentation of system: Good, sphinx readthedocs
  • Nice to haves:
    • Password expiration
    • Import/Export
    • One time view password link
    • Web browser extension


Couldn’t get the docker install to work…

Lyft Confidant

Your secret keeper. Written in python and JS.

  • Support LDAP: No. But has KMS, google and SAML – whatever that means.
  • Self-hosted, open source and not too expensive: Self hosted, open source. No enterprise edition.
  • Level of good security practice: ?
  • Audit trail: Yes, looks pretty basic though
  • Role organsation and management: ?
  • Types of secrets it can store: Seems you can store anything
  • User Experience: Ok – not very intuitive or guiding
  • API existance, level of API documentation and integration: API exists, it is badly documented
  • documentation of system: Home made docs, basics skipped.
  • Nice to haves: CLI tool, one time password link
    • Works closely with AWS IAM
    • Blind credentials



This is a python command line utility that can get ssh keys from your ldap server and add them to your server’s authorized keys.

Documentation on ssh-ldap-pubkey

This tool allowed me to view the SSH public keys for people on ldap.

It was however not automatic…I had to install the package globally. Then I had to manually configure /etc/ldap.conf with the ldap server details.

On a fresh ubuntu you might also have to install:

sudo apt-get install libsasl2-dev python-dev libldap-dev libssl-dev

Another annoying thing is I had to install a specific version 1.3.0 as 1.3.1 was borked.
Then to configure OpenSSH server to fetch users’ authorized keys  you had to add 2 lines to /etc/ssh/sshd_config:

AuthorizedKeysCommand /usr/bin/ssh-ldap-pubkey-wrapper
AuthorizedKeysCommandUser nobody

Still didn’t manage to get it to work

What happens in these cases:

  1. A new user tries to ssh to your server with their ldap username?
  2. A user whose ssh key has been added to your server is removed or taken out of the group on ldap?


A video you can watch on the topic:



Managing LXC/LXD Linux Containers with Terraform

Disclaimer: This post will not speak about creating and maintaining LXC clusters and networking.

This post will introduce LXC/LXD and Terraform seperately, as they are independent things. Once you have enough information about what these things are, I will show you how to use them together.


The lightervisor

LXC is linux containters. A System container manager, offering a similar user experience to virtual machines but using linux containers instead.

LXD and LXC, what the heck?

  • LXC – Linux Containers, a userspace interface using the host’s kernel – the underlying technology
  • LXD – An extension to LXC adding a Rest API – an alternative to LXC’s tools – the interface to the tech

Containers vs Virtual Machines

Containers leverage the OS and kernel of the host they are on, that is why containers (on linux) can’t run windows. A Virtual machine hypervisor is heavyweight, as they have reserved resources for their own linux kernel and host operating system. Containers should be faster to deploy and you can pack them more tightly (density) onto hardware.

What Makes LXC different from Docker containers?

LXC is designed to run as a full machine OS (clean copy of a Linux distribution or a full appliance) and give the experience of a full machine OS. LXD doesn’t care what is running in the container, docker does. Docker on the other hand is built to run a single application – they are focused on being process based containers.

Testing out LXC Containers

On Ubuntu 18.04 (Bionic Beaver) LXC/LXD is preinstalled. Take note it is a canonical product, red hat seems to have chosen docker and dropped support for some lxc libs.

lxc is the command line tool.

LXC recommends installing zfsutils-linux, so we can use so we can use zfs storage pools.

sudo apt install zfsutils-linux

To initialise you need to add the user to the lxd group.

sudo adduser vagrant lxd

Initialise LXD, you can leave most as default allow remote access so terraform can access it later on:

vagrant@ubuntu-bionic:~$ lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing block device? (yes/no) [default=no]: 
Size in GB of the new loop device (1GB minimum) [default=15GB]: 5
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to be available over the network? (yes/no) [default=no]: yes
Address to bind LXD to (not including port) [default=all]: 
Port to bind LXD to [default=8443]: 
Trust password for new clients: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

View networks, storage:

lxc network list
lxc storage list

Create a container, list all containers and enter the shell of the container (by default it will share the amoutn of memory on the host unless limited):

vagrant@ubuntu-bionic:~$ lxc launch ubuntu:16.04
Creating the container
Container name is: fine-buzzard
Starting fine-buzzard
vagrant@ubuntu-bionic:~$ lxc list
|     NAME     |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
| fine-buzzard | RUNNING | (eth0) | fd42:f4c6:f849:da44:216:3eff:fec5:2368 (eth0) | PERSISTENT | 0         |
lxc exec fine-buzzard -- /bin/bash
vagrant@ubuntu-bionic:~$ lxc exec fine-buzzard -- /bin/bash
root@fine-buzzard:~# free -m
              total        used        free      shared  buff/cache   available
Mem:            985          30         894           6          60         954
Swap:             0           0           0

Download a centos image to the local image repo

lxc image copy images:centos/7 local:

What we have done so far is a manual process of managment. A procedure of commands typed with the API via CLI. We aren’t sure of the current state until we check the config. We don’t know the history of how we got to this state.

Let us look at Terraform…


Terraform is an infrastructure as code tool. All you need to do to know the current state of you infrastructure is to look at the code. You are not limited to virtualisation tools, the power of terraform comes from the various things you can manage with it, using providers. Check the providers maintained by terraform and the providers available created and maintained by the community.

Provider categories:

  • Major cloud providers: AWS, Azure, Oracle, GCP, vSphere and vCloud Director
  • Smaller cloud providers: DigitalOcean, Fastly, Linode, Heroku, Hetzner, OpenStack, OVH
  • Infrastructure software: Kubernetes, Vault, Rancher, Mailgun, Consul
  • Network: Cloudflare, DNS, FortiOS, Akamai
  • Version Control: Github, gitlab and bitcbucket
  • Monitoring: New relic, Datadog, Grafana
  • Database: MySQL, Influx, Postgres
  • Many more…

How is Terraform different from Config Management Tools like Ansible

They are both open source and platform agnostic. They are both agentless, have a decent community. Terraform has not been around as long (2014 vs 2012). Ansible has modules to manage many of the providers mentioned above.
The main difference is that Ansible is procedural – code is step-by-step, how to to achieve some desired end state and Terraform is declarative – code specifies your desired end state.
With Terraform: you are aware of what is already deployed
Eg. Ansible

- ec2:
    count: 10
    image: ami-v1    
    instance_type: t2.micro

vs Terraform

resource "aws_instance" "example" {
  count         = 10
  ami           = "ami-v1"
  instance_type = "t2.micro"

Running the above once will produce similar results. The problem is when you want to change something.

Say you wanted to add another 5 servers, with ansible you would need to figure out that 10 were already deployed and then change the count to 5. If you just changed the 10 to 15, you would end up with 25 servers.

Whereas with terraform, it already knows about the 10. So if you change the count to 15, only 5 will be deployed.

With procedural tools, the state of the infrastructure is not fully captured in the code

Ansible allows for mutable infrastructure whereas Terraform forces it to be immutable.

With ansible a history of changes will be built up, ansible will log in via ssh and install something – leading to configuration drift – each server becoming slightly different from another.

With terraform every change to configuration will (usually) return a new object – a new server. You can force terraform to do updates in-place but it is not the natural way. The downside of this is things are deleted – so if you have code or infrastructure with important data on it…it’ll be wiped so need to be very careful.

Terraform Concepts

Terraform has *.tf files that specify the configuration describing your infrastructure. It’s own configuration language, although json can be used.

It is HashiCorp recommended practice that credentials never be hardcoded into *.tf configuration files

Use terraform init to get the correct provider/s.

Then terraform plan to see what will be done on the infrastructure (created, deleted and changed), terraform apply will run plan anyway then ask you to approve.

State is saved in terraform.tfstate or *.tfstate files, so ensure to keep these in version control or use a remote backend.

View current state withterraform show

Remote State Storage

In production environments it is more responsible to share responsibility for infrastructure

The best way to do this is by running Terraform in a remote environment with shared access to state.

Read more about backends in the terraform docs

Terraform with LXD in Practice

Disclaimer: The Terraform-lxd-provider is not maintained by Hashicorp, it is a community provider.

Install Terraform, it is a binary file.

Then install the terraform-lxd-provider from the releases page and then copy that to ~/.terraform.d/plugins on the control host (host running terraform).

Initial setup

You need lxd installed on the managed host, that user needs to be part of the lxd group.

Also install sudo apt install zfsutils-linux

Remember that a default profile resource is installed, called default.

Also ensure that lxd is allowing remote connections with:

    lxc config set core.https_address "[::]"
    lxc config set core.trust_password pass

Provisioning an Infrastructure

On your control host, run terraform init

Lets create our configuration:

# provider to connect to infrastructure
provider "lxd" {
  generate_client_certificates = true
  accept_remote_certificate    = true

  lxd_remote {
    name     = "lxd-server-1"
    scheme   = "https"
    address  = ""
    password = "pass"
    default  = true

# image resources
resource "lxd_cached_image" "ubuntu1804" {
  source_remote = "ubuntu"
  source_image = "18.04"

resource "lxd_cached_image" "centos7" {
  source_remote = "images"
  source_image  = "centos/7"

# containers
resource "lxd_container" "first" {
  config     = {}
  ephemeral  = false
  limits     = {
      "memory" = "128MB"
      "cpu" = 1
  name       = "first"
  profiles   = [
  image      = "${lxd_cached_image.ubuntu1804.fingerprint}"
  wait_for_network = false

resource "lxd_container" "second" {
  config     = {}
  ephemeral  = false
  limits     = {
      "memory" = "128MB"
      "cpu" = 1
  name       = "second"
  profiles   = [
  image      = "${lxd_cached_image.centos7.fingerprint}"
  wait_for_network = false

# Profile
resource "lxd_profile" "terraform_default" {
    config      = {}
    description = "Default LXD profile created by terrraform"
    name        = "terraform_default"

    device {
        name       = "root"
        properties = {
            "path" = "/"
            "pool" = "${}"
        type       = "disk"
    device {
        name       = "eth0"
        properties = {
            "nictype" = "bridged"
            "parent" = "${}"
        type       = "nic"

# Storage Pool
resource "lxd_storage_pool" "default" {
  config = {
      "size"   = "5GB"
      "source" = "/var/lib/lxd/disks/default.img"
      "zfs.pool_name" = "default"
  driver = "zfs"
  name   = "default"

# Bridge Network
resource "lxd_network" "lxdbr0" {
  name = "lxdbr0"
  description = "bridge interface for all containers to access hosts internet"

  config = {
    "ipv4.address" = ""
    "ipv4.nat"     = "true"
    "ipv6.address" = "fd42:38a5:c677:b741::1/64"
    "ipv6.nat"     = "true"

output "second-ip" {
  value = lxd_container.second.ip_address

output "first-ip" {
  value = lxd_container.first.ip_address

Do terraform apply, and hopefully if all goes well your infrastructure will be provisioned.

A terraform.tfstate file will now be created.

Deleting a resource with terraform

How would you delete a resource in terraform?

Simply remove the resource from your configuration – along with any other reference.

Then run terraform apply, which will give you this output saying the container will be destroyed:

  # lxd_container.first will be destroyed
  - resource "lxd_container" "first" {
      - config           = {} -> null
      - ephemeral        = false -> null
      - id               = "first" -> null
      - image            = "368bb7174b679ece9bd0dfe2ab953c02c47ff4451736cb255655ba8348f17bc0" -> null
      - ip_address       = "" -> null
      - limits           = {
          - "cpu"    = "1"
          - "memory" = "128MB"
        } -> null
      - mac_address      = "00:16:3e:eb:d7:3c" -> null
      - name             = "first" -> null
      - privileged       = false -> null
      - profiles         = [
          - "terraform_default",
        ] -> null
      - status           = "Running" -> null
      - wait_for_network = false -> null

The terraform status will have been updated.

Deleting an existing resource outside terraform – within lxc

Lets remove the second container from lxc.

lxc stop second
lxc delete second

Now from terraform run, terraform apply. This will first run a terraform refresh which will update local state file against real resources.
Terraform will now update the local state file that the second container is no longer present.

It will say it will create the resource again:

  # lxd_container.second will be created
  + resource "lxd_container" "second" {
      + ephemeral        = false
      + id               = (known after apply)
      + image            = "368bb7174b679ece9bd0dfe2ab953c02c47ff4451736cb255655ba8348f17bc0"
      + ip_address       = (known after apply)
      + limits           = {
          + "cpu"    = "1"
          + "memory" = "128MB"
      + mac_address      = (known after apply)
      + name             = "second"
      + privileged       = false
      + profiles         = [
          + "terraform_default",
      + status           = (known after apply)
      + wait_for_network = false

Lets apply this change…

Modifying an existing resource with Terraform

Say we want to change a container from ubuntu to centos, and change another container to limit to 512 mb of RAM. Make the following changes in your configuration file:

resource "lxd_container" "first" {
  config     = {}
  ephemeral  = false
  limits     = {
      "memory" = "128MB"
      "cpu" = 1
  name       = "first"
  profiles   = [
  image      = "${lxd_cached_image.centos7.fingerprint}"
  wait_for_network = false

resource "lxd_container" "second" {
  config     = {}
  ephemeral  = false
  limits     = {
      "memory" = "512MB"
      "cpu" = 1
  name       = "second"
  profiles   = [
  image      = "${lxd_cached_image.ubuntu1804.fingerprint}"
  wait_for_network = false

After a terraform apply, you will see:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # lxd_container.first must be replaced
-/+ resource "lxd_container" "first" {
      - config           = {} -> null
        ephemeral        = false
      ~ id               = "first" -> (known after apply)
      ~ image            = "368bb7174b679ece9bd0dfe2ab953c02c47ff4451736cb255655ba8348f17bc0" -> "8ccb053e1872c799616f0eb3534994629192b43e7362df5d0ac5a91772836830" # forces replacement
      ~ ip_address       = "" -> (known after apply)
        limits           = {
            "cpu"    = "1"
            "memory" = "128MB"
      ~ mac_address      = "00:16:3e:17:03:71" -> (known after apply)
        name             = "first"
        privileged       = false
        profiles         = [
      ~ status           = "Running" -> (known after apply)
        wait_for_network = false

  # lxd_container.second will be updated in-place
  ~ resource "lxd_container" "second" {
        config           = {}
        ephemeral        = false
        id               = "second"
        image            = "368bb7174b679ece9bd0dfe2ab953c02c47ff4451736cb255655ba8348f17bc0"
        ip_address       = ""
      ~ limits           = {
            "cpu"    = "1"
          ~ "memory" = "128MB" -> "512MB"
        mac_address      = "00:16:3e:24:cc:62"
        name             = "second"
        privileged       = false
        profiles         = [
        status           = "Running"
        wait_for_network = false

What is hapening here is that the first container is going to be replaced – deleted and then recreated. Whereas the second container will be updated in place – all things done on the vm/container will be maintained. So be very careful as you certainly don’t want to delete important things…

Modifying an existing resource in LXC

Lets rename a resource in LXC

lxc stop first
lxc rename first my-django-app
lxc start my-django-app

The first container no longer exists, so when running a terraform apply…it will state that it is creating a new container called first. Furthermore, it does not know about the renaming and does not import the renamed container into state or config.

Terraform will therefore not manage that resource. In order to ensure terraform does manage that resource you need to import it. That can be done by modifying the config with the name change and doing a terraform import.

resource "lxd_container" "my-django-app" {
  config     = {}
  ephemeral  = false
  limits     = {
      "memory" = "128MB"
      "cpu" = 1
  name       = "my-django-app"
  profiles   = [
  image      = "${lxd_cached_image.centos7.fingerprint}"
  wait_for_network = false

then do:

terraform import my-django-app

The problem is that thisis not done automatically, you have to know about the change made. So if things are managed outside of terraform the state in terraform state files eventually start differing / drifting from config managed directly in the product/service you are managing.

Adding a new resource outside of terraform

What happens when we directly create a new resource in LXC:

lxc launch ubuntu:16.04 -p terraform_default

We are setting it yo use the profile created from terraform as the default profile has no root device.

Terraform will not know about the new image downloaded or the new device.

A problem also is that you cannot import images using this provider – some things just can’t be done but this depends on the provider. This is a community provider.

Error: resource lxd_cached_image doesn’t support import

We can however import the container, provided we know it’s name by creating the resource in config:

resource "lxd_container" "precise-jennet"{


$ terraform import lxd_container.precise-jennet precise-jennet
lxd_container.precise-jennet: Importing from ID "precise-jennet"...
lxd_container.precise-jennet: Import prepared!
Prepared lxd_container for import
lxd_container.precise-jennet: Refreshing state... [id=precise-jennet]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

We aren’t done yet, now as we should copy the config of the imported resource from status into config – annoying that terraform does not do this.

You can do a terraform show and then copy the data over into your config, be sure to exlude things that should be there – although this is trail and error…

resource "lxd_container" "precise-jennet" {
    config      = {}
    ephemeral   = false
    id          = "precise-jennet"
    ip_address  = ""
    limits      = {}
    mac_address = "00:16:3e:bb:09:5e"
    name        = "precise-jennet"
    privileged  = false
    profiles    = [
    status      = "Running"

Extra Sauce

Graphing your infrastructure

brew install graphviz
terraform graph -type=plan | dot -Tpng > plan_graph.png


  • You don’t need to know the steps and order in provisioning, just know the resources and configuration. Terraform will do the rest.
  • Ability to view how infrastrcuture has changed over time
  • Know the current state or blueprint of your infrastructure with 1 command or 1 file view
  • No need to read through the platform/product api docs and concern yourself with the edge cases
  • No need for a cmdb (milage may vary), if the infrastructure is configured as code and checked into a git repo. That is your source of truth.


  • Rolling, zero-downtime deployment, are hard to express in purely declarative terms
  • Difficult creating generic, reusable code.
  • Terraform is not mature yet. It is 0.12.0, so in active dev and might be better to wait for v1
  • Importing must be done one at a time – manually. Does not import config into *.tf config failes
  • Things are deleted and recreated – for things like changing a vm name. Care needs to be taken.
  • Community providers may not be maintained to the level of those maintained by Hashicorp


Horses for courses.


Internet Exchange Resources

NAP Africa – Non-Profit Internet Exchange view who is peering

PeeringDB search for Teraco data centres – You can view information about the data centre, view the local internet exchanges and networks – autonomous system number – given a unique ASN is allocated to each AS for use in BGP routing, they control large blocks of IP addresses.

View the total Number of active IP’s on the network of South Africa, to determine who are the big players.

At 01 August 2019:

AS37353 Seacom Western Cape (Pty) Ltd 3,830,016
AS16637 MTN SA 3,774,464
AS3741 Internet Solutions 2,779,648
AS37168 Cell C (Pty) Ltd 2,752,512
AS5713 Telkom SA Ltd. 2,223,872
AS29975 Vodacom 2,034,432
AS10474 Dimension Data (Pty) Ltd – Optinet 1,443,584
AS37457 Telkom SA Ltd. 1,383,168
AS2018 TENET (The UNINET Project) 1,339,904
AS36937 Liquid Telecommunications South Africa (Pty) Ltd 1,054,720
AS37251 Telkom SA Ltd. 469,504
AS328065 Vast Networks (PTY) LTD 463,104
AS37130 SITA Pty LTD 462,592
AS37611 Afrihost (Pty) Ltd 436,224
AS36994 Vodacom 419,328
AS11845 Vox Telecom Ltd 291,328
AS37105 Neology (Pty) Ltd 207,104