Sending Emails Asynchronously with Django-Celery-Email and RabbitMQ

I am using django-registration-redux to register and activate users. It sends emails to activate and set password. Unfortunately when sending emails it waits for a successful email to send and blocks the main thread so requests takes quite a while. We want all email to send through celery and don’t want to have to create specific tasks so we make use of django-celery-email

Apparently you can use celery to asynchronously send the messages.

Celery requires a broker to send and receive messages, so you have a choice:

  • rabbitmq
  • redis
  • AmazonSQS

We will use rabbitmq. Installing this on ubuntu right off the bat looks looks tricky. You need erlang of a specific version and to app apt-repositories to get a latest version.

I’m going to keep my life simple and


sudo apt install rabbitmq-server

Apparently if you do this on ubuntu 16.04, rabbit-mq is already running in the background, but I didn’t see this message:


Starting rabbitmq-server: SUCCESS

Creating the Celery File

Create a celery.py file in your project root. If you are using a special settings/config location you can put it there next to wsgi.py.


import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mysite.settings')

app = Celery('mysite')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

This code is taken from SimpleIsBetterthanComplex RabbitMQ and Celery Setup. You then need to ensure that celery is imported when django starts in the projects __init__.py:


from .celery import app as celery_app

__all__ = ['celery_app']

Install Django Celery Email

pip install django-celery-email

This will install the django-celery-email package along with the celery dependencies.

Then add this to your settings file: settings/staging.py:


# Celery email sending

CELERY_BROKER_URL = 'amqp://localhost'

INSTALLED_APPS += (
    'djcelery_email',
    'django_celery_results'
)

CELERY_EMAIL_TASK_CONFIG = {
    'name': 'djcelery_email_send',
    'ignore_result': False,
}

CELERY_RESULT_BACKEND = 'django-db'

EMAIL_BACKEND = 'djcelery_email.backends.CeleryEmailBackend'
Migrate to create the message queue database requirements. I’ve also set the CELERY_RESULT_BACKEND so that you can view the result of an AsyncResult.
Migrate: ./manage.py migrate

Getting Things to Work

For the messaging queue system to work rabbitmq-server and celery need to be running.

rabbitmq-server usually creates a system d task and starts on boot up. Celery which was installed with pip will have to be daemonized (celery).

You can test it out by running it directly:

Where config is the project name

celery -A config worker -l info

On a production system with ubuntu 16.04 you can create a daemon:

Create a .celery_env file in the project root:


DJANGO_SETTINGS_MODULE=config.settings.staging

# Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/var/www/django_project/env/bin/celery"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="config"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_LOG_FILE="/var/www/django_project/log/celery.log"
CELERYD_LOG_LEVEL="INFO"

Then create the system.d task:

sudo vim /etc/systemd/system/celery.service

With the following content:


[Unit]
Description=celery service
After=network.target

[Service]
Type=forking
User=staging
Group=www-data
EnvironmentFile=-/var/www/kid_hr/.celery_env
WorkingDirectory=/var/www/kid_hr
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

Finally we need to ensure that the service starts after reboot:

sudo systemctl enable celery.service

Issues

If things aren’t working try sending an email from the django-shell then if you get this error:


Task is waiting for execution or unknown. Any task id that is not known is implied to be in the pending state

you need to ensure the celery service has started, also ensure that you have restarted the entire box.

Otherwise check the log files for any issues.

Sources:

 

Functional Tests with Jenkins, Selenium, GeckoDriver and Firefox

I’ve being going through the django testing goat book, specifically the section on continuous integration. Running you functional tests on jenkins can be quite difficult and you’ll need to tinker around and read the logs to figure out whats what. Here are a few tips.

What is the optimal versions?

Well Jenkins should be the latest because vulnerabilities are always being discovered. As for the rest there are specific compatibility issues between them but I have found the best thing to do is use this rule of thumb: Use the latest.

And I mean the absolute stable latest not just the latest for you distribution.

In my case I was on debian 8 which comes with the iceweasel firefox alternative which was at version 52.4 so I went ahead and got geckodriver and an older version of selenium for the tests. Stupid move. Mostly because you want to be up to date with real users.

Best thing to do is head over to geckodriver releases and install the latest. Get selenium 3.7 or latest and the install the latest stable firefox from firefox releases. For me that was 56.0.2.

Make sure that all the binaries are in your path and check --versions

You can install xvfb

Where Do I look when I get error?

You can check the output in the terminal when you run your test or look at geckodriver.log which I have found to be most useful.

So here are a few errors you might see:

NS_ERROR_SOCKET_ADDRESS_IN_USE

geckodriver by default runs on port 2828 and selenium runs on 4444

So they are probably already running and you need to ensure the processes are destroyed when you tearDown in your tests.

To find the process use:

lsof -l | grep 4444

and destroy it with kill -9 <process_id>

Unable to find a matching set of capabilities

selenium.common.exceptions.WebDriverException: Message: Unable to find a matching set of capabilities

This means you are setting the wrong firefox binary, make sure firefox is installed and in the path.

binary = FirefoxBinary('/usr/bin/firefox')

Message: connection refused

selenium.common.exceptions.WebDriverException: Message: connection refused

This error happens when firefox does not have a display so ensure xvfb is installed and you are starting it in jenkins

geckodriver-selenium-firefox-jenkins-errors

Conclusion

So the headless firefox, selenium, geckodriver and jenkins setup can be tedious but once it is setup you will have a much more robust CI pipeline and ensure that you end users can function onthe site.

Finding a Frontend tool: An Inline Smart Datepicker

I am working on a project that requires an inline smart datepicker. So in this post I will show the process of selecting a decent plugin / tool.

Requirements

The Inline smart datepicker is quite simple and simple is better than complex so I want to keep it that way:

  • Must not pop-up, it should be static and inline on the page
  • Ability to change colour/ style / state of certain dates for public holidays and weekends
  • Events can be assigned to certain dates to show warnings to the user
  • Work on at least internet explorer 11
  • Works well and looks good on mobile

It is always good to write the checklist of requirements down. Keep it as small as possible, so you are not distracted by the nice-to-have.

Finding tools

A good starting place is searching on a search engine. You can use multiple similar search terms like: frontend inline datepicker, html datepicker static and best jquery datepickers

You will find the 30 best datepicker sites and such which help alot. Another good resource is github, just type in datepicker and order by stars. Making sure to avoid the android and iOS options, for that make sure you choose javascript

Now it is just a task of going through the demos and documentation and seeing if the datepickers meet the criteria above.

Shortlist

Shortlist some options that meet your criteria:

  • Zebra Datepicker – Meets all the requirements. Can put custom classes on dates and has events for dates
  • Jquery Datepicker – Meets all the requirements. Can add custom css class name to a date cell and a tooltip which would provide a good extra for informing users about why a date is coloured that way with beforeShowDay
  • Zobuto Datepicker – Meets all the requirements. Has a legend of colours on the chart for more specific info. Has modal popups which is useful for public holidays and leave days for example.
  • glDatepicker – Meets all requirements.
  • Air Datepicker – Meets all requirements. Supports custom cell content and events.

Some that missed:

  • FlatPicker – Colouring and styling individual dates
  • Vue-js Datepicker – Demo broken
  • PickMeUp – Too simple, no colouring
  • PersianDatepicker – For specific language…Persian
  • Datepicker for Bootstrap – Documentation lacking so no individual style elements
  • FullCalendar.io – Overkill for requirements
  • app-datepicker component – Looks nice, but no docs for events and stying individual dates
  • clndr – Styling individual dates can’t be done
  • Rome – Inability to style individual dates
  • Datepicker – has muted class which can act as public holiday, weekends and leave days. It does have events on clicking a date like most do. Does not fully meet styling requirements.
  • e-calendar – Insufficient docs and demos

Selecting the best of the bunch

Now that you have a shortlist take some time reading through the docs and start playing with the libraries.

It is time to choose the extension or plugin with the best docs and the best in your opinion.

Zobuto

I liked zobuto because it worked with and needed resources already in my project bootstrap and jquery. The problem is that it is not an input element…so you have to ensure a date is initialised and linked to an input element in a form. Also initializing the date becomes a bit tricky and it not something I want to explicitly handle.

Also weekends can only be coloured with the data attribute.

Otherwise it works really well. So I have added it to the project in another branch and want to try some of the others out.

It also looks a lot better than some of the others.

 

zabuto-datepicker

Zebra Datepicker

Is limited to the bootstrap style and can’t really display richer information. You can add styling to weekends and it works more like an input but feels a bit more limited.

zebra-datepicker-always-shown

glDatepicker

Doesn’t work well with bootstrap.

gldatepicker-bootstrap-problem