Author: stephen

Adding Time taken to respond to a request in the header of a Django Rest Framework Response. A suitable place to use a Mixin or not?

Ever wanted to add the time taken for a response to your API, so the client knows how long that takes?
I first noticed this cool feature on AWX, a management platform for ansible playbooks.

There’s response headers looked like this:

awx-api-timed-response

So I checked out their source code and copied how they did it:

In generics.py the package extends the base DRF ApiView class and adds a bit of stuff, the important stuff for us is:


    def finalize_response(self, request, response, *args, **kwargs):
        ...
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        response['X-API-Node'] = settings.CLUSTER_HOST_ID
        ...
        
    def initialize_request(self, request, *args, **kwargs):
        ...
        self.time_started = time.time()
        ...

So I created a class extending for ApiView:


class APIView(views.APIView):
    '''
    Add timing to the base APIView class
    '''
    def initialize_request(self, request, *args, **kwargs):
        self.time_started = time.time()
        return super().initialize_request(request, *args, **kwargs)
        
    def finalize_response(self, request, response, *args, **kwargs):
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        if time_started:
            time_elapsed = time.time() - self.time_started
            response['X-API-Time'] = '%0.3fs' % time_elapsed
        return response

Now all I needed to do was entend from my_package.ApiView instead of restframework.views.ApiView and I would get an X-API-Time header.

It worked well for a while, but then I needed to use more Generic Class Based Views (Which extend from ApiView)…check out CDRF. So I could just add these methods, to the extended versions of restframework.generics.ListCreateView etc.

But that just doesn’t feel right. Simply because I can see I am repeating myself multiple times. I just be able to define this functionality once and have all similar objects (those that extend from ApiView) have that functionality…

Using a Mixin to add the Time taken to Respond

I just created a mixin, by extracting those functions into a single class that inherits from nothing:


class TimedAPIMixin:
    def initialize_request(self, request, *args, **kwargs):
        self.time_started = time.time()
        return super().initialize_request(request, *args, **kwargs)

    def finalize_response(self, request, response, *args, **kwargs):
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        if time_started:
            time_elapsed = time.time() - self.time_started
            response['X-API-Time'] = '%0.3fs' % time_elapsed
        return response

Using them in the other views:

 

 


from rest_framework import views
from rest_framework import generics


class APIView(TimedAPIMixin, views.APIView):
    pass


class ListCreateAPIView(TimedAPIMixin, generics.ListCreateAPIView):
    pass


class ListAPIView(TimedAPIMixin, generics.ListAPIView):
    pass

It only works if the mixin is the first thing it inherits from, otherwise the first class will take preference.

Fail2Ban Custom filters and Testing Regex’s against existing Logs

Fail2ban is a tool that can automatically ban malicious bots trying to get into your server. Provided you set up filters and the ip address (or fqhn) is logged you can use fail2ban with any application.

Create a filter, using a regular expression (fail2ban is built with python):

In /etc/fail2ban/filter.d/my-custom-filter.conf:


[Definition]

failregex = ^www.example.com  -.* "POST \/user\/register HTTP\/1.0" 200

ignoreregex =

Now you want to test this for matches against a log file. Ensure that the log file has existing matches.

Make use of the command line tool fail2ban-regex:

fail2ban-regex /var/log/apache2/example-access.log /etc/fail2ban/filter.d/my-custom-filter.conf

You will get summary data like this:


Running tests
=============

Use   failregex file : /etc/fail2ban/filter.d/my-custom-filter.conf
Use         log file : /var/log/apache2/example-access.log


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] ^www.example.com  -.* "POST \/user\/register HTTP\/1.0" 200
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [23636] Day/MONTH/Year:Hour:Minute:Second
`-

Lines: 23636 lines, 0 ignored, 1 matched, 23635 missed
Missed line(s): too many to print.  Use --print-all-missed to print all 23635 lines

Which lets you debug your filter regular expression to ensure it is matching the malicious log entries.

 

Finally, add the new filter to your jail.local by appending the following:


[my-custom-filter]
enabled  = true
filter   = my-custom-filter
action   = iptables-multiport[name=NoAuthFailures, port="http,https"]
logpath  = /var/log/apache2/example-access.log
banTime  = 864000
findtime = 1800
maxRetry = 3

You can read more about the configuration of jails in the manual

Getting Jenkins to deploy with ansible using SSH Agent Forwarding

Your CI/CD tool needs access to code and server, for linting, testing and deploying.
Setup up access on the various devices in a secure manner can be very time consuming. It is important to make use of available technology to make our lives easier.

Jenkins needs access

You will have created credentials for Jenkins – by creating a SSH key pair for your jenkins user. Ensure that that public key have access to the code on your version control platform (as a deploy key).

Now jenkins will be able to get your code and run various tests on it. The problem now is deployment.

jenkins-credentials

Use Jenkins’s SSH credentials to Deploy

We are using ansible to deploy (from the jenkins box). So now jenkins needs access to wherever you are deploying the code to. You would do an ssh-copy-id to get it there.

But there is another problem, when ansible runs the git module to clone the repo you will get an error that the user does not have access.

Correct, the user on the box you are deploying to does not have access to the code. Now you could add that box as another deploy key but now when scaling out to many boxes you will have a hell of alot of ssh credentials to manage.

The best thing to do is use the jenkins user’s cerentials that log into your target box to get the code. This is done with SSH Agent forwarding.

The first thing we will need is theĀ  jenkins SSH agent plugin.

Then enable the SSH agent for your job:

enable-jenkins-ssh-agent

 

Then install the Jenkins ansible plugin and configure it.

Finally you need to tell ansible to use SSH Agent forwarding, otherwise it just won’t do it:


mkdir /etc/ansible
vi /etc/ansiible/ansible.cfg

Add the following config there:


defaults]
host_key_checking = False

[ssh_connection]
ssh_args = -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s

Of course, it is better to ensure host_key_checking is done.

Now everything should work.

Source: SSH Agent Forwarding with Ansible