Author: stephen

Trying the Pelican Static Site Generator and Abandoning it

I have a Jekyll site called fixes

I wanted to move away from Ruby and try the language I prefer and saw that pelican was the most popular. I also wanted to move away to improve the way categories are shown and to speed up the site.

Don’t Use Pelican

After trying it out I have come to the conclusion that it is best to avoid it.

The problems I had:

  • When setting up it creates a fab file for you, for the convenience of deploying and running the site locally. Unfortunately it is based off of the old fabric that required python 2. Using fabric > 2 gave errors.
  • It is September 2018, the last release of Pelican was in January 2017. The author is relying on the community to review pull requests and it seems the community is not alive and kicking anymore.
  • The default theme is not what I deem elegant or simple.
  • AWS is used by default to show assets (like the github ribbon), which is slower and less reliable than a locally hosted version in my opinion. I don’t want to be reliant on some external service.
  • It also uses Google fonts for the default font. Yet more bloat and reliance on Google.
  • Themes are in a seperate repo and are complicated to use.
  • I found that 50% of the themes give errors when you are building them
  • All themes require significant customization.
  • Extra stuff is in yet another repo called pelican plugins, which also requires more technical skill and doc reading to make work.
  • None of the themes I tried were light weight and looked descent and required not much effort.

All in all, things have just gotten a bit too complex and complicated. Too much unnecessary shit is included in the standard package and too much necessary stuff requires external repos and customisation.

Things don’t just work, so I am going to look for another solution or stick with Jekyll.

Authenticated Functional Tests with Selenium and Django

In the Test Driven Development Book for Python and Django by Harry Percival called Obey The Testing Goat, there is a chapter about enhancing the functional test base class and adding pre-authentication so you don’t need to login via the login screen with Selenium.

It uses a custom Email Authentication Backend, but I needed to implement this on a standard: django.contrib.auth.backends.ModelBackend.

My First Attempt


    def create_pre_authenticated_session(self, user):
        '''Create an authenticated user quickly'''
        session = SessionStore()
        session[SESSION_KEY] = user.pk
        session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
        session.save()
        # visit domain (404 quickest)
        self.browser.get(self.live_server_url + "/404_no_such_url/")
        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
        ))

I ran my functional test and something weird was happening, the cookie was getting killed right after this method is called an going to any page.

So I compared based on the cookie itself compared to one that existed in firefox developer tools.

The difference was the httpOnly thingy. So I added it…


        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
            secure=False,
            httpOnly=True
        ))

Nothing changed, the cookie was still gone.

So then I compared an existing decoded session with the one created via the method above.

To find the decoded session:

$ python manage.py shell
[...]
In [1]: from django.contrib.sessions.models import Session

# substitute your session id from your browser cookie here
In [2]: session = Session.objects.get(
    session_key="8u0pygdy9blo696g3n4o078ygt6l8y0y"
)

In [3]: print(session.get_decoded())
{'_auth_user_id': 'obeythetestinggoat@gmail.com', '_auth_user_backend':
'accounts.authentication.PasswordlessAuthenticationBackend'}

The Session Difference

I noticed there was a difference a working session looked like this:

{'_auth_user_id': '1', '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend', '_auth_user_hash': '6a34097f6dab2a1fc68f262e9e67186d2ad5ba93'}

whereas the one I created looked like this:

{'_auth_user_id': 1, '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend'}

So the _auth_user_hash was a problem. I search the django source and found it in auth.

So I set the hash session key with: session[HASH_SESSION_KEY] = user.get_session_auth_hash()

It then worked.

The Solution


    def create_pre_authenticated_session(self, user):
        '''Create an authenticated user quickly'''
        session = SessionStore()
        session[SESSION_KEY] = user.pk
        session[BACKEND_SESSION_KEY] = settings.AUTHENTICATION_BACKENDS[0]
        session[HASH_SESSION_KEY] = user.get_session_auth_hash()
        session.save()
        # visit domain (404 quickest)
        self.browser.get(self.live_server_url + "/404_no_such_url/")
        self.browser.add_cookie(dict(
            name=settings.SESSION_COOKIE_NAME,
            value=session.session_key,
            path='/',
            secure=False,
            httpOnly=True
        ))

How to get data from tables in mysql database into a postgres schema?

I’ve had to do a bit of learning and tinkering to figure out how to do this. I now understand why ETL tools and people that do data migrations and translations are needed. I even will allow use of proprietary GUI based ETL tools.

Anyway the backstory is I created a django project initially for MySQL but needed to change to Postgres later on. I had been using a mysql instance and wanted to migrate the data to the new postgres db.

Converting to Postgres from MySQL

The best tool to use, in my opinion, for a striaght conversion of a source db to postgres is pg_loader.

On the source server I had to install postgres for ease of not having to setup a remote connection and then it is a case of:

sudo apt install pg_loader

Then create the target db in postgres and migrate


$ createdb f1db
$ pgloader mysql://root@localhost/f1db pgsql:///f1db

So we now have a postgres DB which would be suitable for most people.

But I already had a new db instance running and needed to import the data.

Dumping the Postgres DB

An important thing to remember here is that the db already running has a publicschema. This db also has a public schema. Another thing is moving data between databases is  difficult because with postgres you can only connect to one database at a time.

So we should rename the schema in the db we have just converted to something else.


alter schema public rename to original_public;
create schema public;

Then we can import / restore it into the same database we have already running and can copy data between the schemas easily.

Dumping the DB

Use pg_dump to  export just the new schema:


pg_dump -F c -b -v -f --schema=original_public mydb.backup mydb

Restoring the DB

We need to transport the dump to the new server with scp (not in this tutorial).

Once it is accessible, restore the table into the existing db:

pg_restore --dbname=instance_db mydb.backup

Now we have both schemas in the same database.

Moving data between tables

If the tables have the same structure we can easily copy data from one schema to another with a query.


psql
insert into public.the_table select * from original_public.the_table;

and that is that.

Serial Sequencing

There was however an issue when it came to the auto increment fields in my case just primary keys. They were not set at the correct value and the pp broke because of this.

So I manually went and check which number it should be on and updated the relevant fields because I did not have the knowledge.

The knowledge I am referring to is a management command that does this for you called sqlsequencereset

This would have saved me 10 minutes.