Category: DevOps

Authenticated Functional Tests with Selenium and Django

In the Test Driven Development Book for Python and Django by Harry Percival called Obey The Testing Goat, there is a chapter about enhancing the functional test base class and adding pre-authentication so you don’t need to login via the login screen with Selenium.

It uses a custom Email Authentication Backend, but I needed to implement this on a standard: django.contrib.auth.backends.ModelBackend.

My First Attempt


    def create_pre_authenticated_session(self, user):
        '''Create an authenticated user quickly'''
        session = SessionStore()
        session[SESSION_KEY] = user.pk
        session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
        session.save()
        # visit domain (404 quickest)
        self.browser.get(self.live_server_url + "/404_no_such_url/")
        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
        ))

I ran my functional test and something weird was happening, the cookie was getting killed right after this method is called an going to any page.

So I compared based on the cookie itself compared to one that existed in firefox developer tools.

The difference was the httpOnly thingy. So I added it…


        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
            secure=False,
            httpOnly=True
        ))

Nothing changed, the cookie was still gone.

So then I compared an existing decoded session with the one created via the method above.

To find the decoded session:

$ python manage.py shell
[...]
In [1]: from django.contrib.sessions.models import Session

# substitute your session id from your browser cookie here
In [2]: session = Session.objects.get(
    session_key="8u0pygdy9blo696g3n4o078ygt6l8y0y"
)

In [3]: print(session.get_decoded())
{'_auth_user_id': 'obeythetestinggoat@gmail.com', '_auth_user_backend':
'accounts.authentication.PasswordlessAuthenticationBackend'}

The Session Difference

I noticed there was a difference a working session looked like this:

{'_auth_user_id': '1', '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend', '_auth_user_hash': '6a34097f6dab2a1fc68f262e9e67186d2ad5ba93'}

whereas the one I created looked like this:

{'_auth_user_id': 1, '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend'}

So the _auth_user_hash was a problem. I search the django source and found it in auth.

So I set the hash session key with: session[HASH_SESSION_KEY] = user.get_session_auth_hash()

It then worked.

The Solution


    def create_pre_authenticated_session(self, user):
        '''Create an authenticated user quickly'''
        session = SessionStore()
        session[SESSION_KEY] = user.pk
        session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
        session[HASH_SESSION_KEY] = user.get_session_auth_hash()
        session.save()
        # visit domain (404 quickest)
        self.browser.get(self.live_server_url + "/404_no_such_url/")
        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
            secure=False,
            httpOnly=True
        ))

Deploying a django app with dedicated web and db servers

One of the many architectural decisions that will start to impact you when you get to a level where you need to scale is splitting you db and app. Typically we start on a budget and have to share resources but ideally you want to start out separate. The reasons is that the db server will know exactly how much RAM is available to it at all times and will hence improve the consistency and reliability.

Provision 2 Servers

To start off provision 2 (ubuntu) servers, to label things give each a fully qualified domain name like web.myserver.com and db.myserver.com

Then do a basic security and authentication setup on both servers.

The App Server

To setup the app server you can use this guide which uses python 3.6, Nginx, gunicorn and mysql. Just skip the database setup part.

The Database Server

Install postgres.

We need a role (user) for the database and because this role will be adding extensions it needs to be a superuser.

CREATE ROLE dbuser LOGIN PASSWORD 'mydbpass' SUPERUSER;

Importantly we need to look at django’s optimal postgres config

ALTER ROLE dbuser SET client_encoding TO 'utf8';
ALTER ROLE dbuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE dbuser SET timezone TO 'UTC';

Then create the database:

CREATE DATABASE myproject;

Ok…so now fill out the DATABASES setting in your application and make sure the HOST is the internal ip as the servers are within the same data-center hopefully.

But we will need to configure postgres to allow and listen for connections from the internal network. We don’t want public ip’s to have access to it only our other app server within the same datacentre. I’ve done this with MySQL but forgot how to it, so I’m searching how to do it now.

First thing is setup the uncomplicated firewall with:


sudo ufw enable
sudo ufw allow OpenSSH
sudo ufw status

Now we want to enable connections from our app server:

sudo ufw allow from app_server_internal_ip_address to any port 5432

Log into psql and set it to listen on all ip’s:

ALTER SYSTEM SET listen_addresses = '*';

then reload the server:

SELECT pg_reload_conf();

Check where your pg_hba.conf is with:

SELECT name, setting FROM pg_settings WHERE category = 'File Locations';

then add the following line:


# IPv4 local connections:
host    all             all             10.0.0.4/32            md5

Restart


sudo systemctl restart postgresql

Test with the postgres client on the app server:

sudo apt install postgresql-client

There are a few performance tweaks you can do, but I’m always inclined to leave it standard before doing that.

https://www.digitalocean.com/community/tutorials/how-to-secure-postgresql-against-automated-attacks

Allow remote connections to PostgreSQL

https://stackoverflow.com/questions/22080307/access-postgresql-server-from-lan

Setting up a new Macbook for development

A mac is setup for the default user and usually requires a few things to make it into a development machine. It is certain that a fresh ubuntu install is more developer focused than a macbook. In this post I will walk you though what I do when setting up a new macbook for development.

  1. Install homebrew
  2. Install wget
  3. Install python with brew
  4. Download vs code
  5. Download the python extension for vscode
  6. Make tanner terminal the default

Then to improve on all bash (terminal), git and vim prompts etc we are going to use nicolas’s dotfiles, but to use that we have to install xcode from the appstore first. Then you may need to follow this stackoverflow answer if you have previously just installed xcode-command-line tools

Issues with the dotfiles setup

A few issues after installing the dotfiles is:

  • The annoying doink sound when pressing esc and then : in vim
  • The prompt is just showing the current location (it does not show currently logged in user and domain.
  • When moving the cursor or backspacing, if you hold it down for half a second half the characters will be skipped or deleted.

The annoying doink sound when pressing esc and then : in vim

You change this sound in settings -> sound -> alert sound

Cursor Backspace Issue

To fix the cursor and crazy fast backspace issue:


defaults write NSGlobalDomain ApplePressAndHoldEnabled -bool true
defaults write NSGlobalDomain InitialKeyRepeat -int 25
defaults write NSGlobalDomain KeyRepeat -int 6

Changing Prompt colour

Ensure to install the tanner terminal and make sure it is the default terminal theme.

default-tanner-terminal

Incorrect Prompt

The prompt is not as I expected

It is showing:

necolas-dotfile-prompt

Folders and files with different permissions are not different colours

Try ls to have different colour output

I added an alias to ~/.bash_profile

alias ls='ls -G'

Relative paths are tab-completed to absolute paths

 

Terminal not exiting

Make the terminal exit when you type exit and avoid this annoying exit that doesn’t actually exit:

mac-terminal-annoying-exit-that-does-not-exit

You can fix this by changing the terminal settings

make-terminal-close-with-exit