Category: django

Wrangling with Django’s DecimalField and representations

I’ve been building an application that needs precicion as it is dealing with money, finance and trading. For these tasks there is a special datatype called decimal.Decimal that python has built in. It has many advantages but has a few disadvantages when displaying the decimal to the user.

  1. The trailing zeroes are shown (which is unnatural to the normal person)
  2. Sometimes python decides to show the simplist form in scientific notation

let’s get into it…

Python’s Decimal

Working with Decimal is quite nice. They are intialised with string representations of numbers and can be cast to str:


In [1]: from decimal import Decimal

In [2]: precise_number = Decimal('54.899721')

In [3]: precise_number
Out[3]: Decimal('54.899721')

In [4]: str(precise_number)
Out[4]: '54.899721'

Which all seems normal but maybe this won’t:


In [5]: zeroed_num = Decimal('54.000000')

In [6]: zeroed_num
Out[6]: Decimal('54.000000')

In [7]: str(zeroed_num)
Out[7]: '54.000000'

More on this later…

Django’s Decimal Field

Django’s Decimal Field (model field) is slightly different as now we are integrating the saving of this python decimal in a database. That is why we need to set max_digits and decimal_places. So you need to think about the maximum precision expected and maximum total digits in a decimal stored you will be working with in your system. No need to go overboard and from my experience this field can be modified to go bigger.


planned_quantity = models.DecimalField(
        max_digits=19, decimal_places=10,
        blank=True, null=True
    )

So with the above example I can store 999,999,999.1010101010 but nothing more precise or with more digits.

Everything feels great.

Trailing Zeroes

Now after creating a record and now returning to edit the record the form will output the number including the trailing zeroes:

decimal-field-trailing-zeroes

Which just feels wrong. It should just show 3 like a normal human would expect. We saw this a bit earlier and I did a bit of digging. After searching django remove trailing zeroes the result mentioned normalize() which is a function of Decimal:


In [7]: str(zeroed_num)
Out[7]: '54.000000'

In [8]: str(zeroed_num.normalize())
Out[8]: '54'

In [9]: zeroed_num.normalize()
Out[9]: Decimal('54')

As mentioned in the normalize docs it doesn’t just strip the trailing zeroes it also converts 0 to scientific notation.

So it will exhibit this behaviour:


In [11]: damn_decimal = Decimal('600')

In [12]: damn_decimal
Out[12]: Decimal('600')

In [13]: damn_decimal.normalize()
Out[13]: Decimal('6E+2')

In [14]: str(damn_decimal.normalize())
Out[14]: '6E+2'

So normalize is not what we want.

Scientific Notation

So as we alluded to previously sometimes things will just look wierd to the non-scientific on our website:

django-scientific-error-message

This error message is created by the validator:


planned_quantity = models.DecimalField(
        max_digits=19, decimal_places=10,
        blank=True, null=True,
        validators=[MinValueValidator(Decimal('0.0000000001'))]
    )

So things are looking a bit tricky and difficult to work with. Even more so when you bring in validation messages and positive decimal validation.

For now though let us just try and display decimals nicely to the user

The Fix

So we know that normalize is not going to cut it. We’ll need a bit more and that comes from format.

So the intial data of the form will need to be modified for the user and to do that when loading the form do:


    for key in form.initial.keys():
        if isinstance(form.initial[key], decimal.Decimal):
            form.initial[key] = '{0:f}'.format(form.initial[key].normalize())

The important thing here is getting all our decimal fields and then normalizing them.

Then displaying the number in a fixed-point float format with: x = '{:f}'.format(my_dec.normalize())

So that is it. There is a better way of doing this I think and that would be modifying the model and I’m looking at that now.

 

Running multiple django sites on a single server with gunicorn and nginx

When days are dark, friends are few, cash is low and we need to take our django sites and put them on a single server.

I am assuming you have site up and running that is using django with nginx and gunicorn. Now we need to do a few modifications to get the new site working.

First things first get the application code on the site, create your database, create your virtual environment, install your requirements, set any environment variables, migrate, collect static and create your super user. It is that easy (any issues see the guide above)

Creating 2 Gunicorn Services

We setup the nginx config using unix sockets instead of a local http port. So we used:


proxy_pass http://unix:/var/www/myfolder/mysocket.sock;

Instead of:


proxy_pass http://127.0.0.1:8000;

We then created a systemd service to start and stop guncorn for our site called /etc/systemd/system/gunicorn.service

What we need to do now is create a service for each site

Lets rename /etc/systemd/system/gunicorn.service to /etc/systemd/system/site1.gunicorn.service

Make a copy of this site and make relevant changes for the new folder location and socket etc. For a refresher of how this looks:


[Unit]
Description=gunicorn daemon
After=network.target
[Service]
User=
Group=www-data
WorkingDirectory=/var/www/
ExecStart=/var/www//env/bin/gunicorn --access-logfile - --workers 3 --bind unix:/var/www//.sock config.wsgi:application
EnvironmentFile=/var/www//.gunicorn_env
[Install]
WantedBy=multi-user.target

Creating another Nginx server block Config

Make a copy of your existing server config and the new config will look similar to the below, remember to set different log files now as we are hosting multiple sites on a single server.


server {
    listen 80;
    server_name site1.co.za www.site1.co.za;
    error_log /var/log/nginx/site1.error.log
    access_log /var/log/nginx/site1.access.log

    location = /favicon.ico {
        access_log off;
        log_not_found off;
    }   

    location /static/ {
        alias /var/www/site1/site1/staticfiles/;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/var/www/site1/site1.sock;
    }
}

Now to test if things are working:

  1. Reload the daemon with sudo systemctl daemon-reload
  2. Check the status of your new services sudo systemctl status site1.gunicorn.service
  3. Start the services: sudo systemctl start site1.gunicorn.service

Remember to ensure your .guncorn_env file is in the project folder

Now we need to enable the nginx config:


sudo ln -s /etc/nginx/sites-available/site1 /etc/nginx/sites-enabled/

Check you don’t have any types with: nginx -t

Then reload the service: sudo service nginx reload

Now everything should be working correctly.

I would like to thank Khophi for his recent post that pretty much guided me on the things to do to make this work in his post on running multiple django projects behind an nginx proxy

Going Live with Django, MySQL on Ubuntu and server going down

Ever got to deploy day only to have errors streaming in about the site not being able to be accesses. Most likely in your logs or error emails / alerts you get something like this:

(2003, "Can't connect to MySQL server on '127.0.0.1' (111)")

What the hell does this mean, it means MySQL is down and if it was running it means it has crashed. Whoah but if you check the site now it is working…that means nothing the website is unstable as certain actions make it go down. We need to achieve system stability.

The most common reason for MySQL going down is that it is using more memory than physical memory can allocate it then uses virtual memory and crashes soon after that.

What can I do?

First double check the error logs and see if there is a configuration error causing this issue. If there is no clear warning you want to check memory usage.

Check if you are using virtual memory with vmstat


procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 499228  72040 358540    0    0    16    27  230  280  2  1 97  0  0

Check the amount of memory being used:


ps aux | grep mysqld

A few good commands to use during the process:

In mySQL:

show global status;

Source:

Best Practices for Configuring Optimal MySQL Memory Usage