Requests' Two APIs

Kenneth Reitz’s excellent Requests library has been praised, rightfully, for its excellent API. In fact, its API is so good that it’s been praised in a literary context, as well as by almost every programmer who has come across it. There is no question that this API is one of the best you can find in the Python world. It has even inspired other programmers’ API design.

What a lot of people seem not to know, however, is that Requests actually has two APIs. The first API is the highly-praised procedural and object-oriented API that drives making and interacting with HTTP traffic. The second API appeared when Kenneth undertook the giant refactor for v1.0. This is Requests’ barely heralded inheritance-based API.

Requests’ Inheritance API

To discuss the second, super secret Requests API we need to draw one very import distinction. This distinction is between properties of a single HTTP request (things like data, headers and cookies) and things that are properties of the connection (things like the kinds of SSL you can use, or what IP address you send data from).

Requests by and large exposes the first kind of property on its ‘standard’ API. You can set the headers, and the body data, and the cookies, all through the arguments to the Request object and as properties of the Session object. This makes sense: this is the kind of thing that is really specific to a single request or session of requests.

But what about the second kind of thing? Where do you set that? Some of the most-used stuff bubbles its way up to the main API so that it’s easily accessible, but most of it is hardcoded.

I can hear some of you now. “Kenneth hardcoded these options? How awful!” In the Requests versions v0.14.2 and earlier, I’d have probably agreed with you. They aren’t options people want often, but it would be good to have access to them.

The problem is: how do you give access to these options without cluttering the main API so much your documentation looks like this?

Configuration Through Inheritance

If you look carefully at the Requests code, you’ll notice that there are two major objects involved in each request. The first is the Requests Session object (requests.Session). The second is the Requests default Transport Adapter, the HTTPAdapter (requests.adapters.HTTPAdapter). These two objects maintain all of the configuration in Requests.

The Session object is almost entirely concerned with the first kind of property. You can set default headers and default cookies, and that all goes really well. But what if you want to handle the second kind? The answer is surprisingly simple: subclass the HTTPAdapter.

The HTTPAdapter manages all of the connection-level behaviour, and doesn’t expose much in the way of options on its constructor. The real way to affect this behaviour is to subclass it and create your own.

Examples

I’ve shown this configuration once before on this blog, but I want to provide a few examples of what I mean. The first is one I’ve shown before: selecting an SSL version.

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager

class SSLAdapter(HTTPAdapter):
    '''An HTTPS transport adapter that uses an arbitrary SSL version.'''
    def __init__(self, ssl_version=None, **kwargs):
        self.ssl_version = ssl_version
        super(SSLAdapter, self).__init__(**kwargs)

    def init_poolmanaager(self, connections, maxsize):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       ssl_version=self.ssl_version)

You can unconditionally sign requests:

from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager

class SigningAdapter(HTTPAdapter):
    """This adapter signs a request using some unknown method."""
    def send(self, request, **kwargs):
        signature = signing_function(request.body)
        request.headers['Signature'] = signature

        super(SigningAdapter, self).send(request, **kwargs)

You can even support totally different protocols (though having done that myself, I don’t entirely recommend it, but it can be done).

Why This Works

The reason this works is because Requests allows you to configure the backend, using Session.mount(). Before you could do this, anything that was not contained in the Session object was totally inaccessible to users. This meant if you wanted to change the sending logic, you had to subclass the Session and get your hands dirty in the 2000-line send function.

Now, you get to subclass the clearly laid out HTTPAdapter and have to rewrite as little as possible. More people need to know about this and use it, because it’s the best way to really take control of Requests.

Requests has never been more modular or powerful, and it’s API is just as clean as it was when it was born. Take advantage of this secondary API, and you’ll find that you can configure Requests just as easily as you can drive it.