Lukasa's Echochamber

Writing A Transport Adapter

Posted on 29/12/2012.

Last post I briefly mentioned that Python Requests went v1.0. This involved a huge code refactor and a few changes in the API.

One of these changes was the inclusion of something Kenneth (and others) have been thinking about for a while as part of the future of Python HTTP 'project'. This something is the 'Transport Adapter'. From Kenneth's blog post:

Transport adapters will provide a mechanism to define interaction methods for an "HTTP" service. They will allow you to fully mock a web service to fit your needs.

Well, v1.0 brought the inclusion of Transport Adapters into Requests. This is a shiny new feature, and I wanted to take some time to explore it. The best way to do that is to go ahead and write an adapter! As well as providing a useful bit of experience for me, I would be able to document the process and thereby provide a bit of a tutorial for anyone who wants to follow in my footsteps.

But what to write an adapter for? I wanted to see how far Kenneth's goal of Transport Adapters could be stretched, so I decided to implement a transport adapter not just for a specific web service, but for a whole new protocol: FTP.

Come with me, as I show you how to write an adapter through the lens of my own.

For those who just want to see my work, you can see the finished product here.

Getting Started

The first thing to do is to work out how you implement such a thing! A quick look at the Requests code should tell you the answer. A Transport Adapter should be a subclass of requests.adapters.BaseAdapter, and should implement three methods: __init__(), send() and close(). Let's take a closer look at these.

First, __init__(). This will be called only once, when your adapter is instantiated. This means that any state that should be common to all requests belongs here. As an example, Requests' included HTTP Transport Adapter creates a urllib3 ConnectionPool object here. In my case, all I wanted to do was allow multiple FTP verbs, so I created an ad-hoc function table keyed off the names of the verbs. Pretty hacky, but fast.

Second, close(). This is also only called once, at the exact opposite moment to __init__(). Any state you set up in init() should be torn down here. Again, for Requests' built-in adapters, this means tearing down their ConnectionPools. For me, nothing.

Finally, send(). This is where the meat of your functionality will occur, so let's take some time over it.

Send

Glancing through the Requests code reveals that the send() function must, at minimum, accept a single PreparedRequest object and a list of kwargs, and return a Response object. This is where I began to run into trouble.

By default, the PreparedRequest object is very HTTP oriented. This is by design, and any transport adapter you write is likely to be happy with that. However, for me it was mildly problematic. In particular, the PreparedRequest contains almost none of the information on the standard Request object. It carries a url, a method, its headers, its body and its hooks. That's it.

You get a little more detail from the kwargs. You get the values of the following Requests fields: stream, timeout, verify, cert and proxies. Mostly this stuff is of minimal value, and I ended up ignoring it (though I may go back and use timeout.)

If you're using HTTP, I suspect you can mostly get by with this. Byte-level manipulation of the body is possible (and encouraged), so you can permute the Request into whatever form is expected by your web service.

For FTP, this was harder. Mostly I wasn't interested in the data here, except when running an FTP STOR. Here, I wanted to piggyback on Requests' current file upload syntax. However, by the time I get hold of the Prepared Request, the file has been encoded as multipart form-data. My current solution is to use the Python CGI module to decode the data again. This is shoddy, and far and away the worst part of the code I've written.

Returning to your code, this function has all of the responsibility for both sending the data out and parsing it back into a useful form for Requests. You may find, as I did, that you actually want to split this out into several functions or methods.

When building a response, it's particularly instructive to take a look at the function build_response in requests.adapters. This gives you an idea of the kinds of fields Requests expects to be filled in. I ended up basing my equivalent function very heavily on this.

Surprises

There are a few surprises to be found. The first is that response.content and response.text only work if you provide a file-like object as r.raw. I ended up using BytesIO for this, which is a pain. If you're using urllib3 this should be easy for you, but it's worth knowing.

Also bear in mind that you may have to jump through some hoops to get very Requests-like behaviour. The special case of Basic Auth as a naked tuple is handled above the PreparedRequest layer, so if you want to piggyback on it (like I did) be prepared to mess about with headers to get it to work.

Plugging it in

Once you've got a transport adapter, all you need to do is plug it in. To do this, use the mount method of the Session object:

s = requests.Session()
s.mount('http://randomservice', YourAdapter())

In my case, this was actually

s = requests.Session()
s.mount('ftp://', FTPAdapter())

Any subsequent Request through that session will use your transport adapter if the URL prefix matches!

Going Deep

For my stuff, though, I wanted to go a bit further. In particular, since I was adding new verbs (LIST, STOR, RETR and NLST), I wanted to make the utility methods available: the equivalents of Session.get() and Session.post(). To do this required that I monkeypatch the Session object.

I didn't want to do this on import, though: the end user should have the choice about whether they actually want me to mangle the Session object or not. For this reason, I export a function: monkeypatch_session(). This function adds four utility functions to the Session object and overrides the constructor so that the FTPAdapter is automatically mounted.

All Together Now!

When you put that all together, you get my shiny new library. You can get this from the cheeseshop: just run pip install requests-ftp. Once you've got it, you can go straight into using it! For example, you can list the files at the root of the directory and then get one:

import requests
import requests_ftp as ftp
ftp.monkeypatch_session()

with requests.Session() as s:
    r = s.list('ftp://127.0.0.1/', auth=('username', 'password'))
    file = r.content.split('\n')[0]
    r = s.retr('ftp://127.0.0.1/' + file, auth=('username', 'password'))
    file = open('temp.txt', 'wb')
    file.write(r.content)

Caveats

There are lots of things my code doesn't do. A list can be found on the readme. If you feel like you want to use it anyway, feel free! I'd also welcome contributions to resolve some of its more significant problems (like the fact it isn't tested).

Summing Up

Writing a transport adapter for HTTP is very easy, especially as the in-built Requests ones have very clean and comprehensible code. Writing them for new protocols is harder.

This is because Requests is laser-focused on HTTP, as it should be. However, you can take my word for it: it is definitely possible to implement some other protocols as Transport Adapters in Requests. Whether you want to is entirely up to you, of course.

comments powered by Disqus