One of these changes was the inclusion of something Kenneth (and others) have been thinking about for a while as part of the future of Python HTTP ‘project’. This something is the ‘Transport Adapter’. From Kenneth’s blog post:
Transport adapters will provide a mechanism to define interaction methods for an “HTTP” service. They will allow you to fully mock a web service to fit your needs.
Well, v1.0 brought the inclusion of Transport Adapters into Requests. This is a shiny new feature, and I wanted to take some time to explore it. The best way to do that is to go ahead and write an adapter! As well as providing a useful bit of experience for me, I would be able to document the process and thereby provide a bit of a tutorial for anyone who wants to follow in my footsteps.
But what to write an adapter for? I wanted to see how far Kenneth’s goal of Transport Adapters could be stretched, so I decided to implement a transport adapter not just for a specific web service, but for a whole new protocol: FTP.
Come with me, as I show you how to write an adapter through the lens of my own.
For those who just want to see my work, you can see the finished product here.
The first thing to do is to work out how you implement such a thing! A quick
look at the Requests code should tell you the answer. A Transport Adapter
should be a subclass of
requests.adapters.BaseAdapter, and should implement
close(). Let’s take a closer look
__init__(). This will be called only once, when your adapter is
instantiated. This means that any state that should be common to all requests
belongs here. As an example, Requests’ included HTTP Transport Adapter
ConnectionPool object here. In my case, all I wanted to
do was allow multiple FTP verbs, so I created an ad-hoc function table keyed
off the names of the verbs. Pretty hacky, but fast.
close(). This is also only called once, at the exact opposite
__init__(). Any state you set up in
init() should be torn down
here. Again, for Requests’ built-in adapters, this means tearing down their
ConnectionPools. For me, nothing.
send(). This is where the meat of your functionality will occur, so
let’s take some time over it.
Glancing through the Requests code reveals that the
send() function must, at
minimum, accept a single
PreparedRequest object and a list of
Response object. This is where I began to run into trouble.
By default, the
PreparedRequest object is very HTTP oriented. This is by
design, and any transport adapter you write is likely to be happy with that.
However, for me it was mildly problematic. In particular, the
contains almost none of the information on the standard
Request object. It
body and its
You get a little more detail from the
kwargs. You get the values of the
following Requests fields:
Mostly this stuff is of minimal value, and I ended up ignoring it (though I may
go back and use
If you’re using HTTP, I suspect you can mostly get by with this. Byte-level manipulation of the body is possible (and encouraged), so you can permute the Request into whatever form is expected by your web service.
For FTP, this was harder. Mostly I wasn’t interested in the data here, except
when running an FTP STOR. Here, I wanted to piggyback on Requests’ current
file upload syntax. However, by the time I get hold of the
the file has been encoded as multipart form-data. My current solution is to
use the Python CGI module to decode the data again. This is shoddy, and far and
away the worst part of the code I’ve written.
Returning to your code, this function has all of the responsibility for both sending the data out and parsing it back into a useful form for Requests. You may find, as I did, that you actually want to split this out into several functions or methods.
When building a response, it’s particularly instructive to take a look at the
requests.adapters. This gives you an idea of the
kinds of fields Requests expects to be filled in. I ended up basing my
equivalent function very heavily on this.
There are a few surprises to be found. The first is that
response.text only work if you provide a file-like object as
r.raw. I ended
up using BytesIO for this, which is a pain. If you’re using
should be easy for you, but it’s worth knowing.
Also bear in mind that you may have to jump through some hoops to get very
Requests-like behaviour. The special case of Basic Auth as a naked tuple is
handled above the
PreparedRequest layer, so if you want to piggyback on it
(like I did) be prepared to mess about with headers to get it to work.
Plugging it in
Once you’ve got a transport adapter, all you need to do is plug it in. To do
this, use the
mount method of the
s = requests.Session() s.mount('http://randomservice', YourAdapter())
In my case, this was actually
s = requests.Session() s.mount('ftp://', FTPAdapter())
Any subsequent Request through that session will use your transport adapter if the URL prefix matches!
For my stuff, though, I wanted to go a bit further. In particular, since I was
adding new verbs (LIST, STOR, RETR and NLST), I wanted to make the utility
methods available: the equivalents of
do this required that I monkeypatch the
I didn’t want to do this on import, though: the end user should have the choice
about whether they actually want me to mangle the
Session object or not. For
this reason, I export a function:
monkeypatch_session(). This function adds
four utility functions to the
Session object and overrides the constructor so
FTPAdapter is automatically mounted.
All Together Now!
When you put that all together, you get my shiny new
library. You can get this from the
cheeseshop: just run
pip install requests-ftp. Once you’ve got it, you can
go straight into using it! For example, you can list the files at the root of
the directory and then get one:
import requests import requests_ftp as ftp ftp.monkeypatch_session() with requests.Session() as s: r = s.list('ftp://127.0.0.1/', auth=('username', 'password')) file = r.content.split('\n') r = s.retr('ftp://127.0.0.1/' + file, auth=('username', 'password')) file = open('temp.txt', 'wb') file.write(r.content)
There are lots of things my code doesn’t do. A list can be found on the readme. If you feel like you want to use it anyway, feel free! I’d also welcome contributions to resolve some of its more significant problems (like the fact it isn’t tested).
Writing a transport adapter for HTTP is very easy, especially as the in-built Requests ones have very clean and comprehensible code. Writing them for new protocols is harder.
This is because Requests is laser-focused on HTTP, as it should be. However, you can take my word for it: it is definitely possible to implement some other protocols as Transport Adapters in Requests. Whether you want to is entirely up to you, of course.