Category: python

Building a rest API client or SDK as a python package

The best way to learn ways of building your API wrapper / SDK is to build it yourself from scratch is to look at ones that already exist. Here is a list of python api wrappers you can checkout and view. Most if not all of them are already python packages.

Here is a good post on how to test and build an API client / wrapper. In the post the author uses vcrpy as a way to replay stored responses, however they do actually happen the first time. To prevent your tests from ever hitting a real API you should look at Responses.

I also found a few generic wrappers for the requests library that are supposed to make your life easier, but it looked like many were just abstraction for abstraction sake and the closer your are to requests the better.

I did however find uplink which is a python api boilerplate module based on retrofit – which is the goto http client for Android development. I like it and I think it will make my life a bit easier when creating the api wrapper. It is however still in beta. Tapioca is another one, but I didn’t rate it. It arised from this fellow not liking how API wrappers are done and the added abstraction.

As a side note, Braintree doesn’t even make the rest API available, it forces clients to use a client or wrapper built by them mainly to ensure security.

I found uplink hard to work with…it seems as though the further you are from requests the harder your life becomes.

Allowing for a User Provided Session

One of the key things needed when making an API wrapper/client available is letting the user provide their own session. A reason for this is that user could have added additional auth to the API they are connecting to or may be using a different method of auth altogether.

Also, the client may be using an API gateway like kong so the client may need an auth token or some other permission. So it is better to just allow the user to provide their own session – which can then be modified with the session id’s.

So in the class constructor – __init__() method, ensure a session can be set and default it to none. The below example is adapted from the hvac client. eg:


class MyClient(object):
    """My Api Client"""

    def __init__(self, base_uri=DEFAULT_BASE_URI, cert=None, verify=True, timeout=30, proxies=None, session=None):
        """Create a new request adapter instance.

        :param base_uri: Base URL for the Vault instance being addressed.
        :type base_uri: str
        :param cert: Certificates for use in requests sent to the Vault instance. This should be a tuple with the
            certificate and then key.
        :type cert: tuple
        :param verify: Either a boolean to indicate whether TLS verification should be performed when sending requests to Vault,
            or a string pointing at the CA bundle to use for verification. See http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification.
        :type verify: Union[bool,str]
        :param timeout: The timeout value for requests sent to Vault.
        :type timeout: int
        :param proxies: Proxies to use when preforming requests.
            See: http://docs.python-requests.org/en/master/user/advanced/#proxies
        :type proxies: dict
        :param session: Optional session object to use when performing request.
        :type session: request.Session
        """
        if not session:
            session = requests.Session()

        self.base_uri = base_uri
        self.token = token
        self.namespace = namespace
        self.session = session
        self.allow_redirects = allow_redirects

        self._kwargs = {
            'cert': cert,
            'verify': verify,
            'timeout': timeout,
            'proxies': proxies,
        }

As you can see in the example above it is not just the session. Other things like SSL verification, the certificate to use for SSL verification and proxies can be passed to the constrator to allow for versatility.

Should the client raise it’s own Errors

If a request on your managed node API returns an error 4xx or 5xx, should your API client raise an exception or just let the response be handled by the caller?

Say you do catch exceptions, should those exceptions be custom created by the client package or just be part of pythons standard lib, such as ValueError.

From what I have seen, many python api clients do raise their own errors stemming from a root exception. According to Brett Slatkin of Effective Python:

Having a root exception lets consumers of your API catch exceptions you raise on purpose

They should help you find bugs, not hide them.

Some examples:

Although you can choose to use the standard libraries excpetions if you want, an example of this would be the DEPRECATED heroku python client wrapper or the new heroku python client wrapper

How to store a session based token that gets Revoked

Different API’s authenticate in different ways. Some will provide you a token that only lasts a certain length of time, thereafter your requests stop being authenticated.

How should we handle this? Get a new token each time – probably not a good idea, store the token in a key value store or store it as a global variable.

Also what mechanism should we use to ensure a new token is requested, when the old one expires?

How to automatically reauth for a specific error code or catch errors

In order to catch these error codes and raise our own, it is best to do it in a single place. Pretty much however the client is making requests – perhaps with the requests library request method. We would have to extend from the request class and do our clients error checking after we get a response – in our custom requests class.

One way of doing that is how hvac used in the adapter class.

The problem is that we are allowing users of the library to provide their own requests.Session() object, that won’t be using our custom request method.

The idea here is to have your client class define it’s own request method and call that method instead of using requests. In other words MyApiClient.get() which behind the scenes uses self.session.get, but adds error checking.

 

Building a Python HTTP client that automatically refreshes access tokens

How often is it that when calling API’s you do an initial round of auth to receive a token that expires after a certain amount of time?
It is quite common these days. What isn’t common is a generic way to ensure that this authentication process is automatically restarted when the existing token expires.

The first thing to remember is that we should use some persistent storage mechanism to keep the token.

Where are we going to store this access token value then? We need to store it as it needs to be accessed between multiple requests.

The database is not a terrible place, I’ve seen it done before. Or you can use django’s cache API.

Finding an Example

I wish there was a client I could view and see how other people have implemented this in python.

Here is an apt comment from the php community:

…for other APIs I have found the decorator pattern around the HTTP client (especially a PSR-18 client, because its interface is so simple) is a great place to put the renewal logic. That’s the point where the expiry response is known, the full original request is known so can be tried again, and the renewed authentication token can be saved in the current client and saved to a persistent storage.

So I understand, here you have a stack of middleware (kind of) handlers that each wrap the request and pass back the response. On an expired token response you try renewing the token, then bounce the same request back down the chain again? The retry count allows you to give up after a number of bounces.

You could inject a closure in this middleware here to handle the persisting of the new token to storage.

JudgeJ comment on guzzle

A Javascript implementation example, the gist:

  1. Store the token
  2. Wrap every request in a decorator that reauths when a `401` is received
  3. Ensure the original request is redone

I managed to find a python implementation for python-fitbit

Storing the Token in Cache

You can store the auth token in cache like so:


from django.core.cache import cache 

@property
def session(self):
    session = requests.Session()
    session.verify = settings.VERIFY_SSL

    # Check there is a token saved
    auth_header = cache.get('auth_header')
    if not auth_header:
        auth_header = self.get_authorization(session)
        cache.set('auth_header', auth_header)

    session.headers.update(auth_header)

    return session

 

Adding Time taken to respond to a request in the header of a Django Rest Framework Response. A suitable place to use a Mixin or not?

Ever wanted to add the time taken for a response to your API, so the client knows how long that takes?
I first noticed this cool feature on AWX, a management platform for ansible playbooks.

There’s response headers looked like this:

awx-api-timed-response

So I checked out their source code and copied how they did it:

In generics.py the package extends the base DRF ApiView class and adds a bit of stuff, the important stuff for us is:


    def finalize_response(self, request, response, *args, **kwargs):
        ...
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        response['X-API-Node'] = settings.CLUSTER_HOST_ID
        ...
        
    def initialize_request(self, request, *args, **kwargs):
        ...
        self.time_started = time.time()
        ...

So I created a class extending for ApiView:


class APIView(views.APIView):
    '''
    Add timing to the base APIView class
    '''
    def initialize_request(self, request, *args, **kwargs):
        self.time_started = time.time()
        return super().initialize_request(request, *args, **kwargs)
        
    def finalize_response(self, request, response, *args, **kwargs):
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        if time_started:
            time_elapsed = time.time() - self.time_started
            response['X-API-Time'] = '%0.3fs' % time_elapsed
        return response

Now all I needed to do was entend from my_package.ApiView instead of restframework.views.ApiView and I would get an X-API-Time header.

It worked well for a while, but then I needed to use more Generic Class Based Views (Which extend from ApiView)…check out CDRF. So I could just add these methods, to the extended versions of restframework.generics.ListCreateView etc.

But that just doesn’t feel right. Simply because I can see I am repeating myself multiple times. I just be able to define this functionality once and have all similar objects (those that extend from ApiView) have that functionality…

Using a Mixin to add the Time taken to Respond

I just created a mixin, by extracting those functions into a single class that inherits from nothing:


class TimedAPIMixin:
    def initialize_request(self, request, *args, **kwargs):
        self.time_started = time.time()
        return super().initialize_request(request, *args, **kwargs)

    def finalize_response(self, request, response, *args, **kwargs):
        response = super().finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
        if time_started:
            time_elapsed = time.time() - self.time_started
            response['X-API-Time'] = '%0.3fs' % time_elapsed
        return response

Using them in the other views:

 

 


from rest_framework import views
from rest_framework import generics


class APIView(TimedAPIMixin, views.APIView):
    pass


class ListCreateAPIView(TimedAPIMixin, generics.ListCreateAPIView):
    pass


class ListAPIView(TimedAPIMixin, generics.ListAPIView):
    pass

It only works if the mixin is the first thing it inherits from, otherwise the first class will take preference.