Category: GNU/Linux

Prerequisite Packages and Compiling Python 3 on CentOS

What are the prerequisite packages for a complete python3 compilation install?

You will always get issues like Pip not being able to access pypi because the openssl module was not installed. Other things need the gcc compiler and such.

Recently I got this warning:

Could not import the lzma module. Your installed Python is incomplete. Attempting to use lzma compression will result in a RuntimeError.

It is very annoying

Install Prerequisites

yum groupinstall development
yum install zlib-devel gcc openssl-devel bzip2-devel libffi-devel xz-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel expat-devel

Compile python

cd /opt
curl -O https://www.python.org/ftp/python/3.8.3/Python-3.8.3.tgz
tar xzf Python-3.8.3.tgz
./configure
make
sudo make install

More Issues

There may be a warning after running make:

The necessary bits to build these optional modules were not found:
_curses               _curses_panel         _dbm               
_gdbm                 _sqlite3              _tkinter           
_uuid                 readline     

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd                
time                                                           

This post mentions that some more are required (which I retrospectively added above):

    sudo yum install yum-utils
    sudo yum groupinstall development
    # Libraries needed during compilation to enable all features of Python:
    sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel

Even after that I still get:

    The following modules found by detect_modules() in setup.py, have been
    built by the Makefile instead, as configured by the Setup files:
    _abc                  atexit                pwd                
    time   

Upgrading SQLite on CentOS to 3.8.3 or Later

Let me guess you are using django and may have just done an upgrade to django 2.2.x or 3.x in order to stay up to date and have the latest security updates.
This means you will need to upgrade SQlite as django only supports SQlite 3.8.3 and later.

Unfortunately CentOS 6/7 only has v3.7.17 of SQLite in their repos.

So you need to install v3.8.3 of SQLite or the latest from source.

To install from source (I'm not sure how to use the precompile binaries):

  1. Download the source code from sqlite downloads

cd /opt
wget https://www.sqlite.org/2019/sqlite-autoconf-3280000.tar.gz
tar -xzf sqlite-autoconf-3280000.tar.gz
cd sqlite-autoconf-3280000
./configure
make
sudo make install

However this doesn't help us, as from the python repl - it is still using the old version:


python
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.7.17'

You need to recompile python to use the new path to sqlite3.

Recompiling Python to Use the New Sqlite3

When compiling sqlite3 you get some useful info (that we usually skip over):

Libraries have been installed in:
   /usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

This is exactly what we want to do - we must use the library directory: /usr/local/lib

Check you sqlite version:

[root@server]# sqlite3 --version
3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6

Then make sure to compile python again using the LD_RUN_PATH environment variable.
It is better to use this variable over LD_LIBRARY_PATH.
Using LD_LIBRARY_PATH- whenever python is run (at runtime) it will look for linked libraries with that path.
What we want is for the libraries to be cooked into python at link time - compile time.

So lets compile python from source which you can get from python.org/downloads:

cd /opt/Python-x.y.z
LD_RUN_PATH=/usr/local/lib  ./configure
LD_RUN_PATH=/usr/local/lib make
LD_RUN_PATH=/usr/local/lib make altinstall

Now you should be using the updated version

[root@server Python-3.6.8]# python3
Python 3.6.8 (default, May  8 2020, 11:34:53) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.sqlite_version
'3.31.1'

More information on the dynamic linker at the gcc docs

If you do not use LD_RUN_PATH, then you have to make sure that the LD_RUN_PATH environment variable is set to /usr/local/lib for every user that is going to run python - which can be really annoying to do.

Hopefully your python environment is now using the correct sqlite version.

Checking SQlite3 version quickly

How to quickly check your SQLite version

python3.6 -c "import sqlite3; print(sqlite3.sqlite_version)"

Sources

Cheapest VPS in South Africa

What company provides the cheapest VPS in South Africa.

These days we all need the cloud (also known as computers that you don't own physically) to host our websites, API's, hold our (or customer) data and run applications.

Recently I needed an elasticsearch application set up to make monitoring and analysing my API gateway easier. I tried to install that on my 1Gb VPS and it complained. Thanks JVM:


There is insufficient memory for the Java Runtime Environment to continue

So I went looking for a cheap Virtual Private Server that is on the Linux platform and based in South Africa (because my clients are in South Africa and latecy will be faster 20 milliseconds as opposed to 200ms when the server is in europe).

I have used both domains.co.za and cloudafrica.net, as they were the cheapest options I had found.

Any extra included services like backup are disregarded. Services that only provide HDD (hard drives and not Solid state drives - SSD's) like afrihost are excluded.

I am going to do a bit of research to try and find a cheaper deal for a virtual private server in South Africa, my finds are below:

CompanyMemory (GB)CPU (Cores)Storage (GB)Price (R/month)
cloudafrica.net4448305
domains.co.za44125498
hostafrica.co.za43100415
vps.co.za42100400
telasera.com4?120399
cloud.co.za42100340
hostking.co.za4250539
web4africa.co.za441501350
web-telecoms.co.za42100399
1-grid.com42100519
paradigmsolutions.co.za42100899

 

If you have found a cheaper deal for servers hosted in South Africa, please let me know.