Category: IAM

Practical Application: Implementing SSH security with TLS certificates

In any organisation of a large size managing access to servers and cloud resources is difficult.
There is often a tradeoff between convenience and security.
Changing these settings is also a bit scary in production as you can be locked out of your servers...

One solution mentioned by facebook engineering and smallstep is to make use of certificates to authenticate and authorise SSH users.

In this post we will look at what we need to achieve better SSH security and how to implement it...

What we need to know

The topics we should read up on are:

  • TLS certificates and public key cryptography
  • OpenSSL
  • Public Key Infrastructure (PKI)
  • SSH
  • Hashicorp Vault

Some books might be:

That is alot of reading.

Sources

Integrating Keycloak and Harbor Registry with OpenID Connect

The documentation for setting up an OpenIDC identity provider / authentication method for Harbor Registry can be found in the harbor docs.

Harbor has supported OIDC since version 1.8.

Importantly:

You can change the authentication mode from database to OIDC only if no local users have been added to the database. If there is at least one user other than admin in the Harbor database, you cannot change the authentication mode.

So if you have existing local users, you will need to remove them - unfortunately doing this from the admin frontend does not actually delete them. you have to enter the postgres db and delete associated projects and then the users.

Information for setting up the client on keycloak side can be found on the red hat docs page

Getting Started

Having said all that...

  1. As Admin, go to Administration -> Configuration -> Authentication

  2. Select Auth mode as OIDC

  3. Fill in the required information as per the below screenshot:
    harbor-oidc-config-keycloak

  4. Click Test Configuration

OIDC Endpoint

For keycloak you can get your realm's OIDC details by going to:

https://<base_url>/auth/realms/<realm_name>/.well-known/openid-configuration

But for the OIDC configuration you remove everthing up to /.well-known... including the back slash.
So the OIDC endpoint should be:

https://<base_url>/auth/realms/<realm_name>

Deleting Existing Harbor Users

If you are using harbor on kubernetes - you can enter the postgres pod and execute in the shell:

docker exec -it harbor-db bash
psql -U postgres
\c registry
select * from harbor_user
delete from harbor_user where user_id > 2

Source

Setting up Keycloak on Kubernetes

First thing to do is get familiar with keycloak. once you are happy it might be useful take a look at the keycloak quickstarts.
They seem to have all the examples and samples on getting going with keycloak.

In particular you want to look at the keycloak examples

For posterity I will show the contents of keycloak.yaml:

apiVersion: v1
kind: Service
metadata:
  name: keycloak
  labels:
    app: keycloak
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: default
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:10.0.1
        env:
        - name: KEYCLOAK_USER
          value: "admin"
        - name: KEYCLOAK_PASSWORD
          value: "admin"
        - name: PROXY_ADDRESS_FORWARDING
          value: "true"
        ports:
        - name: http
          containerPort: 8080
        - name: https
          containerPort: 8443
        readinessProbe:
          httpGet:
            path: /auth/realms/master
            port: 8080

and keycloak-ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: keycloak
spec:
  tls:
    - hosts:
      - KEYCLOAK_HOST
  rules:
  - host: KEYCLOAK_HOST
    http:
      paths:
      - backend:
          serviceName: keycloak
          servicePort: 8080

Environment Variables

We want to customise a few things about how keycloak runs and we do this by updating the environment variables.
So let us find what environment variabels are available and which we need to change.

We know the image beind used it:

quay.io/keycloak/keycloak:10.0.1

So lets see what the readme of that container image says

It is rather dissapointing that when we check on quay for keycloak, there is an empty readme. So our princess is in another castle.

The best readme I could find was on keycloak-containers.

So the list of available environment variables I could find were:

  • KEYCLOAK_USER
  • KEYCLOAK_PASSWORD
  • DB_VENDOR - h2, postgres, mysql, mariadb, oracle, mssql
  • DB_ADDR - database hostname
  • DB_PORT - optoinal defaults to vendor port
  • DB_DATABASE - database name
  • DB_SCHEMA - only postgres uses this
  • DB_USER - user to auth with db
  • DB_PASSWORD - user password to auth with db
  • KEYCLOAK_FRONTEND_URL - A set fixed url for frontend requests
  • KEYCLOAK_LOGLEVEL
  • ROOT_LOGLEVEL - ALL, DEBUG,ERROR, FATAL, INFO, OFF, TRACE and WARN
  • KEYCLOAK_STATISTICS - db,http or all

Oh I found an even more exhaustive list of environment variables in the docker entrypoint

Creating a K8s service as a reference to an external servie

As per kubernetes up and running, it is worthwhile to represent an external service in kubernetes. That way you get built in naming, service discovery and it looks like the database is a k8s service.

It also helps when replacing a service or switching between prod and test.

my-db.yaml:

kind: Service
apiVersion: v1
metadata:
  name: external-database
  namespace: prod
spec:
  type: ExternalName
  externalName: database.company.com

If you just have an ip you need to create the service and the endpoint with:

kind: Service
apiVersion: v1
metadata:
  name: keycloak-external-db-ip
spec:
  ports:
    - protocol: TCP
      port: 3306
      targetPort: 3306
kind: Endpoints
apiVersion: v1
metadata:
  name: keycloak-external-db-ip
subsets:
  - addresses:
    - ip: my-ip.example.com
    ports:
    - port: 3306

now the actual service dns name will be:

    my-svc.my-namespace.svc.cluster.local

so in this case:

    keycloak-external-db-ip.keycloak.svc.cluster.local

Set that as DB_ADDR with the other credentials and we should be good to go.

So updarte that and the other environment variables and deploy:

Create the deployment:

kubectl create -f keycloak-deployment.yml -n keycloak

create the service and the ingress:

kubectl apply -f keycloak-service.yml -n keycloak
kubectl apply -f keycloak-ingress.yml -n keycloak

Boom and you should be up and running

Sources