Category: Web Development

Deploying a django website to a Ubuntu 16.04 server with python3.6, gunicorn, nginx and Mysql

Getting up and running on your local development setup and being able to build and see the changes you are making is one of the numerous reasons we like django.

The built in development webserver helps a lot in this regard.

As soon as we have to deploy the site on a server and make it public so that other people can see the project and progress the job is more do it yourself and can be hard at times. In this post, I will show you how I did it with Ubuntu 16.04, nginx, gunicorn and with a Mysql Database.

I tried to use an ansible script to build an idempotent setup but it became tedious and decided that the initial server configuration will be a snowflake but change deploys will be done with ansible.

ubutnu16.04-django-nginx-mysql-gunicorn

ubutnu16.04-django-nginx-mysql-gunicorn ubutnu16.04-django-nginx-mysql-gunicorn ubutnu16.04-django-nginx-mysql-gunicorn

 

There are a few topics and a flow of how to get the site deployed:

  1. Provisioning a server
  2. Configuring the server with requirements
  3. Settings
  4. Requirements
  5. Basic Django Setup
  6. Getting gunicorn to work
  7. Configure Nginx to Proxy Pass

Provisioning a Server

I want an ubuntu 16.04 server mainly because it has long term support and I am used to ubuntu. With your hosting client create the server and then ssh in with:

ssh user@123.345.566.789

The best thing to do now is take care of basic security and initial setup of the ubuntu 16.04 server

The most important thing is creating an ssh key and logging in with ssh and disabling password login.

Configuring the server with requirements

One key thing is that when installing python 3.6 on ubuntu 16.04 is that it is not part of that version and was released later.

You should build python3.6 from source and not use a ppa. This tutorial on installing python3.6 from source on ubuntu 16.04 is the one I used.

Everything else can be installed with apt:

  • python-mysqldb
  • mysql-server
  • nginx
  • git
  • libmysqlclient-dev

Remember when installing mysql-server that you will set a root password. You can run mysql_secure_installation to ensure the server is secure.

Settings

In your django project it is important to ensure that you split setting.py into a directory with files

  • settings/base.py
  • settings/local.py
  • settings/staging.py

base.py contains global generic settings then in local.py and staging.py you can import those settings with:

from . base import *
and then override and add settings you need.
Remember that when running a manage.py command you should specify the settings with:
./manage.py collectstatic --setting=config.settings.staging
Although setting an environment variable is better with:
export DJANGO_SETTINGS_MODULE=config.settings.staging
Another thing is typing settings in on your local machine will become tedious so better to default to your local settings in manage.py with:

if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")

Remember to ensure mail servers and external integrations are not production or live settings when they should not be

Requirements

A good idea is to split up your requirements the same way you did for settings.

  • requirements/base.txt
  • requirements/local.txt
  • requirements/production.txt

You can inherit from base with -r base.txt as the first line.

Basic Django Setup

Create the database and user

Now you can create a mysql schema and a mysql user for that database. See this tutorial on how to create a new user and grant permissions.

Create the Environment

python3.6 -m venv env

source env/bin/activate

Now ensure that the settings environment variable is set in the environment by adding the following to the end of env/bin/activate:

export DJANGO_SETTINGS_MODULE=config.settings.staging

deactivate and reactivate with source env/bin/activate

Importantly you can put the django setting above in post_activate only if you use virtualenvwrapper it does not work with the native python virtualenv module.

Setup the database and static files

./manage.py migrate

./manage.py collectstatic

./manage.py createsuperuser

Test the site works

With the ALLOWED_HOSTS = [‘xxx’, ]  and DATABASES updated in settings you can test the site with the development server with:

manage.py runserver 0.0.0.0:8000

You will need to enable the 8000 port first with sudo ufw allow 8000

You can test if the site works at: http://server_domain_or_IP:8000

Getting Gunicorn to Work

Great news. If everything has worked up till now we can now get gunicorn to work.

pip install gunicorn

Make sure to add the to your requirements/production.txt

Run gunicorn:


gunicorn --bind 0.0.0.0:8000 settings_module.wsgi

Now you can test again.

If everything is good we want this service to be managed now by the os, so that it starts automatically on a system start and can be monitored with logs. For that unforunately we need to create a systemd service.

vim /etc/systemd/system/gunicorn.service

Add the following:


[Unit]
Description=gunicorn daemon
After=network.target

[Service]
User=<server_user>
Group=www-data
WorkingDirectory=/var/www/<project_name>
ExecStart=/var/www/<project_name>/env/bin/gunicorn --access-logfile - --workers 3 --bind unix:/var/www/<project_name>/kid_hr.sock config.wsgi:application
EnvironmentFile=/var/www/<project_name>/.gunicorn_env

[Install]
WantedBy=multi-user.target

Important to note that an environment file is given .gunicorn_env

The contents of this file will contain all the environment variables needed so in our case just:

DJANGO_SETTINGS_MODULE=config.settings.staging

You now need to create and enable the service:

sudo systemctl start gunicorn

sudo systemctl enable gunicorn

Configure Nginx to Proxy Pass

The final step is serving the site through nginx

sudo vim /etc/nginx/sites-available/<project_name>

Add the following:


server {
    listen 80;
    server_name <server_domain_or_ip>;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /var/www/<folder_name>;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/var/www/<folder_name>/kid_hr.sock;
    }

}

You then need to create a symlink to the enabled sites folder and remove the default site symlink there.

sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

Check you nginx config with:

sudo nginx -t

then restart nginx

sudo service nginx restart

Open port 80 and delete the old 8000 port rule:

sudo ufw delete allow 8000

sudo ufw allow 'Nginx Full'

Done!

Conclusion

So it is not that difficult but it is difficult if you are trying to create a repeatable thing, which I hope one of you reading this will do. Another thing you can do is add whitenoise for simplified static file serving which I have not tried yet.

If you have any issues you can comment below or troubleshoot on the source of this article

VueJS and Django using them together

If you have been a web developer for some time, eventually you realise you have been wasting a lot of time and doing things wrong (or at least making your life difficult) if you have not been using Django and VueJS.

Django being the backend where the data is stored and where resources are maintained, deleted and created. VueJS being the frontend where relevant data is displayed in a concise and logical way. They both do there respective duties pretty well…in my opinion like a dream.

They also both benefit from a relatively shallow learning curve to start with.

Getting these 2 great frameworks to play well together is an important part of creating a web application that is versatile and a joy to work with.

So without further ado…how do we get them working together

VueJS and Django a quick overview

There is a great presentation given by Andrew McCracken on this topic and a lot of the info in this blog post is derived from that presentation.

Something to keep in mind when thinking in the MVC context: With django the view is really the controller and the template is really the view.

So django already has the template part and VueJS is also a templating library amongst other thing, so there is a bit of overlap. However, if we create a frontend agnostic backend (An API) then they will work together much like how they were designed.

So Django can be used with the Django Rest Framework (DRF) to provide the API and Vue can be used with vue-resource to interact with the API.

Django, Vue and the template

One aspect which jumps right at you is that when displaying a variable from data in a template…both django and vue uses the {{ my_var }}double braces (for a reference on brackets, braces and parenthesis)

Django’s variables will be resolved first and it will break your intended template.

There is a horrible way to get around this that I have used:


{% templatetag openvariable %} build.fields.description {% templatetag closevariable %}

But there is a nicer way to do this by just changing the configuration in vue to use the square brackets instead of braces

In the global configuration add:

Vue.config.delimiters = ["[[", "]]"];

So Vue template tags can now be easily distinguished from django ones.

Other aspects

I am building out a project with a bit more than just including the pre-packaged Vue file and am using webpack to bring in Vue so a few issues will no doubt come up and I will post them

 

Getting Started VueJS

To bootstrap the project execute the following commands


sudo npm install -g vue-init
vue init webpack frontend
cd frontend/
npm install
npm run dev

This install vue-init so you can bootstrap your project which in this case I have called frontend

But now you must be thinking…well node is serving the site wouldn’t django be serving the site in my case. And yes, I think the same way but then maybe we won’t have access to the auto-reload and other features

Now after doing this I was Unsure…

About how to wire up Django and Vue with webpack, so I reached out to the community and Jarrod Hobbs suggested looking at django-webpack-loader

Why is Laravel 5 So Bad?

Coming from Zend 1 and Yii 1, I find Laravel has made things complicated for itself. It has also tried to be too much of a unique item. Let me explain why Laravel 5 is so bad.

why-is-laravel-5-so-bad

Who am I to Comment

Well I’ve used Yii 1 and Zend Framework extensively. You can take a look at my Yii share trading site. I have also dabbled, gone through a video tutorial and built a simple site and API with it, with Ruby on Rails 4. I have also worked with Magento (Sigh) and the Django Rest Framework. More recently I have been developing and learning python and django extensively.

So pretty much the best non-js server-side frameworks. That being said, I thought I would take a look at Taylor Otwells master piece, Laravel 5, before I check out Yii 2 (not going to do this anymore). Unfortunately in some respects I am disappointed (take note I’m only about 20% through an ebook teaching laravel).

The Good of Laravel 5

  • It leverages off other new and cool stuff: embracing npm, task runners, dev environments and more recently the best js frameowrk in my opinion (vuejs)
  • It uses the best (not just the easiest or most popular) out of the “new and cool stuff”
  • Homestead: a great development environment facilitator and cli tool

The Bad of Laravel 5

  • You have to manually create a view and action, no generator
  • laravel and php artisan command line utilities could be combined into a single laravel command
  • The directory structure is strange to the point of complexity. Controllers for example are not found in the app folder but in the app/Http folder. Views are not in the app/Http directory next to controllers, they are in a separate top level folder Resources/view

The Ugly of Laravel 5

    • It wraps everything as its own special thing: homestead is a wrapper for vagrant, elixir is a wrapper for gulp. Just use the tools as they are because it will allow php devs to work on a host of other frameworks and technologies without having to know domain specific commands.
    • Strange View and layout directives (maybe I need to get used to them): @yield, @extends, @section, @show, @parent. It can get rather complicated.
    • The Blade templating system, as above, declaring variables in views makes it seem like you are using angular, react or meteor. Sometimes I think php short tags make things simpler than: @foreach, @endforeach, @if, @else, @elseif, endif.
    • No generic route for Controller/Action/Id, although this is actually ok. Explicit is better than implicit.

Update (26-10-2015)

Another thing to notice is the caching of views. This feature is on by default and you have to hack it to turn off view caching. What this means is you can’t pinpoint in what file an error has occurred (usually a typo). So you end up with a crap error like:

Crap-Laravel-View-Caching-Errors

Summing it Up

Laravel 5 is probably the best PHP Framework. It’s still not great though but it a massive step in the right direction. I can’t help but think that yii2 may have got the rails to PHP conversion done a bit better. You will have to know Laravel 5 though because of this report, so might as well start linking it.

The best feature of Laravel is by far the Standard Local Development Environment: Homestead.
So instead of having to install all these crap PHP extensions and what not you can create a repeatable local laravel dev environment that just works. Furthermore if you use the homestead composer tool, creating shared folders and adding multiple sites to virtual host id a breeze. It makes development much better. Unfortunately frameworks like Django and Rails aren’t as Vagrant friendly and hence you have to create an ansible script to setup your boxes which is a bit of a schlep.

Update 2 (29-10-2015)

There is also the issue of the blade templating system. Which is awesome, except that it is not strict and allows any old crap to pass.

Such as ending a section with @endsection instead of @stop

or specifying a section with:

@section('title', 'Establishment Name')@stop

Instead of:


@section('title')
Establishment Name
@stop

No error messages, and nothing in the logs. However it creates a huge problem in that…the session cookie is not set correctly due to template errors.

So now you will never be able to maintain a logged in session and all forms will now fail with a:

TokenMismatchException in VerifyCsrfToken.php line 46

Crap-Laravel-View-Caching-Errors

laravel CSRF Token Mismatch Excpetion

Also you will find it is very hard to peruse the templates to find where the actual problem is and you will have to resort to deleting large portions of the template until the problem code is found.

The blade templates need a linter, to point out templating errors in the editor. Why Laravel should have one built into elixir.