Category: django

Deploying a django app with dedicated web and db servers

One of the many architectural decisions that will start to impact you when you get to a level where you need to scale is splitting you db and app. Typically we start on a budget and have to share resources but ideally you want to start out separate. The reasons is that the db server will know exactly how much RAM is available to it at all times and will hence improve the consistency and reliability.

Provision 2 Servers

To start off provision 2 (ubuntu) servers, to label things give each a fully qualified domain name like web.myserver.com and db.myserver.com

Then do a basic security and authentication setup on both servers.

The App Server

To setup the app server you can use this guide which uses python 3.6, Nginx, gunicorn and mysql. Just skip the database setup part.

The Database Server

Install postgres.

We need a role (user) for the database and because this role will be adding extensions it needs to be a superuser.

CREATE ROLE dbuser LOGIN PASSWORD 'mydbpass' SUPERUSER;

Importantly we need to look at django's optimal postgres config

ALTER ROLE dbuser SET client_encoding TO 'utf8';
ALTER ROLE dbuser SET default_transaction_isolation TO 'read committed';
ALTER ROLE dbuser SET timezone TO 'UTC';

Then create the database:

CREATE DATABASE myproject;

Ok...so now fill out the DATABASES setting in your application and make sure the HOST is the internal ip as the servers are within the same data-center hopefully.

But we will need to configure postgres to allow and listen for connections from the internal network. We don't want public ip's to have access to it only our other app server within the same datacentre. I've done this with MySQL but forgot how to it, so I'm searching how to do it now.

First thing is setup the uncomplicated firewall with:


sudo ufw enable
sudo ufw allow OpenSSH
sudo ufw status

Now we want to enable connections from our app server:

sudo ufw allow from app_server_internal_ip_address to any port 5432

Log into psql and set it to listen on all ip's:

ALTER SYSTEM SET listen_addresses = '*';

then reload the server:

SELECT pg_reload_conf();

Check where your pg_hba.conf is with:

SELECT name, setting FROM pg_settings WHERE category = 'File Locations';

then add the following line:


# IPv4 local connections:
host    all             all             10.0.0.4/32            md5

Restart


sudo systemctl restart postgresql

Test with the postgres client on the app server:

sudo apt install postgresql-client

There are a few performance tweaks you can do, but I'm always inclined to leave it standard before doing that.

https://www.digitalocean.com/community/tutorials/how-to-secure-postgresql-against-automated-attacks

Allow remote connections to PostgreSQL

https://stackoverflow.com/questions/22080307/access-postgresql-server-from-lan

Functional Tests with Django Tenant Schemas and Selenium

On my journey of adding multi-tenancy to my app, I have used django-tenant-schemas and the testing utilities it provides to make my unit tests work with the tenants or main site.

However the functional tests is another kettle of fish as there is not utility to use for them.

Functional Tests

From the django docs on the topic:

Use in-browser frameworks like Selenium to test rendered HTML and the behavior of Web pages, namely JavaScript functionality. Django also provides special support for those frameworks; see the section on LiveServerTestCase for more details.

LiveServerTestCase is the same as a TransactionTestCase with one added feature in that it launches a live django server in the background for browser based tests and simulations with selenium.

Good to know that the TransactionTestCase differs from a normal TestCase in the way the db is reset after tests. TransactionTestCase will truncate the tables, whereas TestCase will just rollback the transaction.

The live server listens on localhost and binds to port 0 which uses a free port assigned by the operating system. The server’s URL can be accessed with self.live_server_url during the tests.

If you are using the staticfiles app then you need to use StaticLiveServerTestCase to ensure the static files are loaded.

Now what are the issues that I can foresee:

  1. There is a public facing site (with PUBLIC_URLS) and tenant sites that run on different domains. I would need to test both sites with selenium.
  2. The test tenant is not created anymore and I will have to subclass the StaticLiveServerTestsCase to create the test tenant (and public tenant that uses the public schema)

Other than that all is good.

The Public Schema

Functional tests should simulate reality as much as possible. So I want there to be a public schema and at least 2 tenants (Update: nope I don't need to do this, as this should be tested by django_tenant_schemas itself). So in creating the base functional test case, these 3 things need to be created.

The first thing to do is get the setUp of the functional test right. Now regular setup of the app needs to be linked with a tenant db and not public as the public schema will only have the SHARED_APPS and not the TENANT_APPS so we  need to separate it out.

So I started on the public test case, I took everything to its base bones and made a simple request:


self.browser.get(f'{ self.live_server_url }')

Which gave a bad request error. Usually when you get a 400 it is because your public schema is not there. So you will need to create your TenantModel Record and ensure the domain is localhost.

 


class PublicFunctionalTest(BaseFunctionalTest):
    '''Functional test just for the public facing site'''
    def setUp(self):
        '''Create the public schema'''
        super().setUp()
        Client.objects.create(
            domain_url='localhost',
            schema_name='public',
            name='public'
        )

This worked well and I was sent to the public website, made easier due to the default of localhost by the TestCase. Now you can add all your public facing functional tests.

Creating the Tenant Schema

Creating the public schema in the public facing functional test was straightforward because the server in the background goes to localhost and that is the domain_url of the schema we setup.

Now to makes things as simple as possible, I just realised you can set the tenant schema to use localhost and everything should work similarly. There are however a few other things that seem to need to be done based on the provided TenantTestCase which I've pasted here:

 


from django.conf import settings
from django.core.management import call_command
from django.db import connection
from django.test import TestCase
from tenant_schemas.utils import get_public_schema_name, get_tenant_model

ALLOWED_TEST_DOMAIN = '.test.com'


class TenantTestCase(TestCase):
    @classmethod
    def add_allowed_test_domain(cls):
        # ALLOWED_HOSTS is a special setting of Django setup_test_environment so we can't modify it with helpers
        if ALLOWED_TEST_DOMAIN not in settings.ALLOWED_HOSTS:
            settings.ALLOWED_HOSTS += [ALLOWED_TEST_DOMAIN]

    @classmethod
    def remove_allowed_test_domain(cls):
        if ALLOWED_TEST_DOMAIN in settings.ALLOWED_HOSTS:
            settings.ALLOWED_HOSTS.remove(ALLOWED_TEST_DOMAIN)

    @classmethod
    def setUpClass(cls):
        cls.sync_shared()
        cls.add_allowed_test_domain()
        tenant_domain = 'tenant.test.com'
        cls.tenant = get_tenant_model()(domain_url=tenant_domain, schema_name='test')
        cls.tenant.save(verbosity=0)  # todo: is there any way to get the verbosity from the test command here?

        connection.set_tenant(cls.tenant)

    @classmethod
    def tearDownClass(cls):
        connection.set_schema_to_public()
        cls.tenant.delete()

        cls.remove_allowed_test_domain()
        cursor = connection.cursor()
        cursor.execute('DROP SCHEMA IF EXISTS test CASCADE')

    @classmethod
    def sync_shared(cls):
        call_command('migrate_schemas',
                     schema_name=get_public_schema_name(),
                     interactive=False,
                     verbosity=0)


class FastTenantTestCase(TenantTestCase):
    @classmethod
    def setUpClass(cls):
        cls.sync_shared()
        cls.add_allowed_test_domain()
        tenant_domain = 'tenant.test.com'

        TenantModel = get_tenant_model()
        try:
            cls.tenant = TenantModel.objects.get(domain_url=tenant_domain, schema_name='test')
        except:
            cls.tenant = TenantModel(domain_url=tenant_domain, schema_name='test')
            cls.tenant.save(verbosity=0)

        connection.set_tenant(cls.tenant)

    @classmethod
    def tearDownClass(cls):
        connection.set_schema_to_public()
cls.remove_allowed_test_domain()

There are quite a few things being done here, one thing that I don't think will work is setting the domain_url of the tenant to tenant.test.com which would require a modification to the host file.

We would need things like set_tenant  . So we need to decide if it is better to inherit from TenantTestCase and add the functionality of the StaticLiveServer or the other way around. Or perhaps use Multiple Inheritance.

Wait I think I've found an example where a Mixin would be the appropriate choice. So I pretty much copied the TenantTestCase and extended from StaticLiveServerTestCase instead of TestCase.


class StaticLiveServerTenantTestCase(StaticLiveServerTestCase):

However I have a wrapper of the StaticLiveServerTestCase that adds functionality from harry percival's obey the testing goat book that takes screenshots, waits for elements to appear etc.

Now I don't want to duplicate these methods just applied to different classes because they extend from different classes.

 

Converting a Single Serving Django Application into Saas

I built a time tracking, leave application and billing application with django, called crowdminder. The product was created in partnership with a consulting company, they paid me and I developed it for them while retaining co-ownership.

I have seen that they have benefited from this simple application and know that many other consulting companies in South Africa could benefit too, so I want to make it into a Software-as-a-service product.

But where to start...

Dynamic Settings

Tenants should also be able to change settings for their instance. With django's default settings file that would be very difficult and I thought I should take a look at django-constance to see if it may benefit my situation.

Django constance allows you to set some settings as dynamic so they will be retrieved from the database instead of the settings file. For example: TIME_ZONE, TIMECAPTURE_INTERVAL_MINUTES, BILLABLE_HOURS_IN_WORKING_DAY.

I have separated these settings out and it seems to be working quite well so far.

Multi-Tenancy

I realised that to make the product available to many others, it would need to be able to have multiple customers or tenants.

The choice of you multi-tenant strategy needs consideration for your individual case but in my case it made sense to use  django-tenant-schemas.

As it would minimise the change to tables (I would need a tenant linked to every record in the db if I went with a shared db). With the shared schemas approach though I would need to switch to postgres from mysql.

Setup

Importantly there are a few setup steps like explicitly telling schemas which apps are shared and which are tenant specific. Shared apps will create tables in the public schema. I have to admit that you will run into problems and it is better to use a rule of thumb keep things in tenant apps unless they definitely will be shared. An important thing to understand is that django.contrib.auth in both SHARED_APPS and TENANT APPS would allow users to be created for the public site and for each tenant.

Furthermore with django-tenant-schemas you should never use migrate as that will migrate everything into the current schema instead, migrate_schemas should be used. Migrate schemas can be run in parallel if a speed up is needed.

Then after migrating just the public schemas, we need to create a public tenant, whose domain should just be the base domain of the site. Then other tenants can be created.

Management commands that apply to tenants now need to inherit from BaseTenantCommand

Calling management commands will also need added care: eg. /manage.py tenant_command do_foo --schema=customer1

Thrid Party Apps

Celery also needs to be made to use tenants

Issues while working with postgres

django.db.utils.ProgrammingError: cannot cast type numeric to interval
LINE 1: ...R COLUMN "duration" TYPE interval USING "duration"::interval

django.db.utils.OperationalError: cannot DROP TABLE "auth_group_permissions" because it has pending trigger events

It won't even let you delete the database:

postgres-headache-wont-let-delete-database

Another gem from the django docs on postgres migrations:

PostgreSQL is the most capable of all the databases here in terms of schema support; the only caveat is that adding columns with default values will cause a full rewrite of the table, for a time proportional to its size.

For this reason, it’s recommended you always create new columns with null=True, as this way they will be added immediately.

When running tests the initial migrations panicked with:

django.db.utils.ProgrammingError: relation "users_user" does not exist

Testing Multitenancy

Testing the multitenant sites and public site was a bit tricky in some cases. To test the public schema website the PUBLIC_SCHEMA_URLCONF were not being used so I had to override the ROOT_URLCONF to point to the public. Also the public schema was not created automatically so I had to do that in setUp().

Another thing is that I sometimes use WebTest for my tests, as it is sometimes better than the standard Django TestCase. When working with tenant tests, there is a Class that can be used that does a lot of the tenant related stuff for you so converting those tests would look like this:


from django_webtest import WebTestMixin
from tenant_schemas.test.cases import TenantTestCase

class EntryUnitTests(WebTestMixin, TenantTestCase):
    ...
Yet More Issues

Another fucking issue, is that the migrations didn't seem to be running before the tests. The tables were not created for the tenant with TenantTestCase.

The ContentType model needs to be used with care as the model field is saved in camelcase with MySQL but in lowercase with postgres.

So instead of:


leave_content_type = ContentType.objects.get(
app_label="leave", model="LeaveRequest"
)
you should use:

leave_content_type = ContentType.objects.get(
app_label__iexact="leave", model__iexact="LeaveRequest"
)

Damn you postgres:


Type 'yes' if you would like to try deleting the test database 'test_crowdminder', or 'no' to cancel: yes
Destroying old test database for alias 'default'...
Got an error recreating the test database: database "test_crowdminder" is being accessed by other users
DETAIL:  There is 1 other session using the database.

Ok, I just needed to kill the other test that was running after cancelling an ipdb session.

So it looks like django-webtest is not allowing transactions within tests:


    def do_request(self, req, status, expect_errors):
        # Django closes the database connection after every request;
        # this breaks the use of transactions in your tests.
        if close_old_connections is not None:  # Django 1.6+
            signals.request_started.disconnect(close_old_connections)
            signals.request_finished.disconnect(close_old_connections)
        else:  # Django < 1.6
            signals.request_finished.disconnect(close_connection)

        try:
req.environ.setdefault('REMOTE_ADDR', '127.0.0.1')

But that doesn't make sense as the tests passed previously (with MySQL) and django-constance...it turns out if you fix the `HTTP_HOST` so the request happens properly (and doesn't return a 400). If a 400 is returned the current transaction is aborted.

Site URL and Emails

Previously the app send emails from the `DEFAULT_FROM_EMAIL` and sometimes in the template I had to use a custom settings SITE_URL to set urls on emails and templates. Now I envision that each tenant will have a <my_company>.crowdminder.co.za domain, so these settings will need to be dynamic as well (but not editable by the admin user)

Protected Files

I have enabled a way to protect files based on permissions based on the id of the file and if the site is built with an isolated database or schema then there would be duplicated id's and that would affect file retrieveal. So a tenant based folder would need to be created.

For this I may need to look at using the tenant storage API

Potentially Useful Packages when building a Saas

Apart from the packages mentioned above there may be some more that will make it easier to build a multi-tenant Saas platform with Django.

Django-ikari - Anchored sub-domains

Django-guardian - Per-object permissions

Django-billing - Plan based subscription controls

Django-pricing - Plan based subscription definitions

Django-merlin - multi-step django forms

Django-waffle - A feature flipper, controlling which users can see the features so there is not one massive release / conflict during production deployment.

Sources