Author: surfer190

Kong: Name resolution failed

Sometimes when calling a service on kong you might get a response like this

HTTP/1.1 503 Service Temporarily Unavailable
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Thu, 02 Apr 2020 09:28:06 GMT
Server: kong/2.0.1
X-Kong-Response-Latency: 12

{
    "message": "name resolution failed"
}

What does it mean?

Kong cannot access the upstream

Make sure the corret protocol http or https is used.
Also if the upstream's SSL cannot be verified then

How to update, change and delete a storage Profile with VCloud Director REST API

Recently I was tasked with removing a few attached storage profiles from a number of VDC's in vCloud Director. The task would have been a long and arduous manual task.

Fortunately vCloud Director has a REST API (A flippen good one).
The reason I say it is good is that it makes proper used of HTTP verbs to do certains tasks and returns the correct error codes and good error messages.

For example if you make a GET to a resource path that only allows PUT you will get a 415 METHOD NOT ALLOWED. So it helps to guide you - it is self documenting.

The only shit thing is that it is application/xml only right now, but xml isn't that bad to be fair.

You when you make a bad request (400) you will get extra info in the error message - ie. it will say what fields it expected or it didn't expect and you can correct the payload.

Remember the main page for the VCloud Director Rest documentation. I am using vCloud Director 9.5 (Cloud API 31.0 ) .

Changing a Storage Profile

The first thing I reliased is that you cannot just delete a storage profile. The profile must not be the default profile and it must be disabled first.

Unfortunately the documentation only gave examples on how to add or delete a storage profile - no way of updating it.

If you look at the service provider documentation and the Update Organization VDC Storage Profiles docs, you see there is no info about updating a profile.

If you look at the xml for a storage profile you will see it looks like this:

<AdminVdcStorageProfile href="https://{my_vcd_base_url}/api/admin/vdcStorageProfile/{uuid}...
<Link href="https://{my_vcd_base_url}/api/admin/vdcStorageProfile/{uuid}" rel="edit" type="application/vnd.vmware.admin.vdcStorageProfile+xml"/>

You will see there is an edit relation for that url.
After a bit of trial and error I saw that you could send it a PUT with the Storage profile object and it would update:

response = client.session.put(
            f'https://{my_vcd_base_url}/api/admin/vdcStorageProfile/{uuid}',
            data=f'''<?xml version="1.0" encoding="utf-8"?>
                <AdminVdcStorageProfile name="{storage_profile_name}" xmlns="http://www.vmware.com/vcloud/v1.5">
                    <Enabled>false</Enabled>
                    <Units>MB</Units>
                    <Limit>50000</Limit>
                    <Default>false</Default>
                    <ProviderVdcStorageProfile.../>
                </AdminVdcStorageProfile>
            ''',
            headers={
                'content-type': 'application/vnd.vmware.admin.vdcStorageProfile+xml'
            }
        )

that worked

So now I could set a storage profile as disabled and change it from being the default.

Deleting the storage Profile

Now deleting is as per the recent documentation on updating storage profile on the VDC. Importantly you need to add a RemoveStorageProfile element and not a DeleteStorageProfile element as per earlier docs.

Another thing, is the url you are posting to is the vdc's storage profile endpoint and there must be at least an href attribute on the RemoveStorageProfile element - the one you want to remove.

response = client.session.post(
        f'https://{my_vcd_base_url}/api/admin/vdc/{vdc_uuid}/vdcStorageProfiles',
        data=f'''<?xml version="1.0" encoding="UTF-8"?>
            <UpdateVdcStorageProfiles
            xmlns="http://www.vmware.com/vcloud/v1.5" >
                <RemoveStorageProfile href="{storage_profile_url}">
                </RemoveStorageProfile>
            </UpdateVdcStorageProfiles>
        ''',
        headers={
            'content-type': 'application/vnd.vmware.admin.updateVdcStorageProfiles+xml'
        }
    )

The response returns a task.
Which has a URL that you can use to check if the delettion was successful.

That's it. Hopefully this will help to automate some of the tasks you require.

How to speed up http calls in python? With Examples

Blocking HTTP requests

The most popular and easy to use blocking http client is requests. The requests docs are simple and straight forward...for humans.

The biggest performance gain you can acquire (provided you are making requests to a single host) is using an http session. This creates a persistent connection, meaning that additional requests will use the existing session. More info on that in this blog post on python and fast http clients.

Example Blocking HTTP persistent connection vs new connections

Here is some example code for getting quotes from quotes.rest

import requests
import time

def get_sites(sites):
    data = []

    session = requests.Session()

    for site in sites:
        response = session.get(site)
        data.append(response.json())

    return data

if __name__ == '__main__':
    categories = ["inspire", "management", "sports", "life", "funny", "love", "art", "students"]

    sites = [
        f'https://quotes.rest/qod?category={category}' for category in categories
    ]

    start_time = time.time()
    data = get_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")

Don't overuse this API as they have rate limits and will eventually give you a 429 http status code as a response - Too Many Requests

So when I run this code:

Downloaded 8 sites in 3.7488651275634766 seconds

That is pretty fast, but what would be the case if I used requests.get() instead of using the session?

In that case the result was:

Downloaded 8 sites in 10.602024793624878 seconds

So in the first example resusing the existing HTTP connection was 2.8 times faster.

Threaded HTTP Requests

There is a library that uses requests apparently called requests_futures that uses threads - preemptive multithreading.

Example Threaded Request Futures

from concurrent.futures import as_completed
from requests_futures import sessions
import time

def get_sites(sites):
    data = []

    with sessions.FuturesSession() as session:
        futures = [session.get(site) for site in sites]
        for future in as_completed(futures):
            resp = future.result()
            data.append(resp.json())

    return data

if __name__ == '__main__':
    categories = ["inspire", "management", "sports", "life", "funny", "love", "art", "students"]

    sites = [
        f'https://quotes.rest/qod?category={category}' for category in categories
    ]

    start_time = time.time()
    data = get_sites(sites)
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")

When running this code it was faster:

Downloaded 8 sites in 1.4970569610595703 seconds

Interestingly if I set the max workers to 8 sessions.FuturesSession(max_workers=8), it slows it down dramatically;

Downloaded 8 sites in 5.838595867156982 seconds

Anyway the threaded requests is 7 times faster than non-persistent blocking http and 2.5 times fast than persistent blocking http.

Asynchronous HTTP Requests

The next thing to look at is co-operative multitasking, which still uses a single thread (and single process) but will give control of execution back to the event loop once's it is done - it won't block.

Python has a few aync http libraries: aiohttpand httpx

Example Async Aiohttp

from aiohttp import ClientSession
import asyncio
import time

async def get_sites(sites):
    tasks = [asyncio.create_task(fetch_site(s)) for s in sites] 
    return await asyncio.gather(*tasks)  

async def fetch_site(url):
    async with ClientSession() as session:
        async with session.get(url) as resp:  
            data = await resp.json()
    return data

if __name__ == '__main__':
    categories = ["inspire", "management", "sports", "life", "funny", "love", "art", "students"]

    sites = [
        f'https://quotes.rest/qod?category={category}' for category in categories
    ]

    start_time = time.time()
    data = asyncio.run(get_sites(sites))
    duration = time.time() - start_time
    print(f"Downloaded {len(sites)} sites in {duration} seconds")

The result of this code was:

Downloaded 8 sites in 1.271439790725708 seconds

That is the fastest response we have had yet. More than 8 times faster than the non-persistent blocking HTTP connection, almost 3 times faster than the persistent blocking HTTP connection.
Also 17% Faster than the threaded blocking HTTP requests.

Thoughts on Aiohttp

The problem with aiohttp as rogouelynn mentions in a blog post is everything needs to be async

In real life scenarios you often need to do some syncronous stuff first, like authenticating and receiving a token.

You can't just do:

session = ClientSession()
response = session.get('https://iol.co.za') 

>>> response
<aiohttp.client._RequestContextManager at 0x102985c80>

As you only get back a context manager.

Potencially an easier to use library is httpx because syncronous requests are as native and easy to do as asynchronous requests.

r = httpx.get('https://httpbin.org/get')

Putting it all together

How to speed up http calls in python...well go through the steps until you get the speed you need.

  1. Use simple blocking persistent HTTP connections with requests.Session()
  2. Use an Asynchronous http client like asyncio or httpx.

In the steps above I skip over the threading part as you will find that when you scale up threading can become unreliable and it is usually the case where asyn is better or matches threading performance.

python-http-client-speed-comparison