Coming Soon to a Zig Near You: HTTP Client

Fore-forwarning: This post describes std.http efforts pre-0.11 zig. After the 0.11 release new features have been introduced, and some names have been changed, this post does not describe those changes. To attempt to keep this post as a useful guide: if you're using zig 0.12 (or a development version thereof, as 0.12 has not been released as of this edit), apply the following changes:

• Client.request is now Client.open
• Request.start is now Request.send
• Client.fetch has been added as a way to bypass most of the complexity introduced in this post

Forewarning: This post is intended to be a overview of the new features included under std.http. As a result, this post is rather technical, and not a comprehensive guide for everything possible with the new additions.

Zig's standard library has gained increasing support for HTTP/1.1 (starting with Andrew's work on a basic client for use in the official package manager). In a series of pull requests and an assortment of bug fixes std.http has gone from a bare-bones client to a functional client (and a server, but that will be described later) supporting some of the following:

• Connection pooling: It will attempt to reuse an existing keep-alive connection and skip the DNS resolution, TCP handshake and TLS initialization
• Compressed content: It will automatically decompress any content that was enthusiastically compressed by a server.
  • This only handles compression encoding, which is when the server decides to compress a payload before sending it. It does not handle compression that was part of the original payload.
• HTTP proxies: We can tell a client to proxy all of its requests through a HTTP proxy and it will just work.
  • This doesn't support more complicated protocols such as SOCKS or HTTP CONNECT proxies.
• TLS: If you make a request to a HTTPS server, it will automatically form a TLS connection and send the request over that.
  • This is implemented using std.crypto.tls, which is still a work in progress. It only supports a small subset of ciphersuites that are common on the internet, so you may run into issues with some servers. If you get an error.TlsAlert, it's very likely that the server doesn't support any of the ciphersuites Zig implements.

Everything described in this post is available in Zig 0.11.0-dev.2675-0eebc2588 and above.

What this means for Zig (and You!)

If you're not using the master branch of Zig, not much yet. However, these changes will be available in Zig 0.11.0.

While std.http doesn't support the fancy newest protocols (those who have read the HTTP2 specification might understand why), HTTP/1.1 is still by far the most common version of the HTTP suite.

With a fresh install of Zig, we can fetch the contents of a website or send a POST request with no extra hassle of finding an extra library to provide those features for us.

How do I use it?

For the remainder of this post, I will assume you have decided which allocator you want to use and defined it as allocator.

Creating a Client

The first thing we need to do is create a std.http.Client. This can't fail, so all we need to do is initialize it with our allocator.

var client = std.http.Client{
    .allocator = allocator,
};

Making a Request

Now that we have a Client, it will sit there and do nothing unless we tell it to make a request, for that we'll need a few things:

• a std.http.Method
• a std.Uri (which we should parse from a url)
• a std.http.Headers (yes, even if we don't plan on adding anything to it) to hold the headers we'll be sending to the server.
  • if we're avoiding allocations, don't worry: it will only allocate if we append anything to it.

And optionally:

• a buffer for response headers if we want to avoid the Client from dynamically allocating them.

We should obtain a std.Uri by parsing one from a string. This might fail if you give it an invalid URL.

const uri = try std.Uri.parse("https://example.com");

We can initialize a std.http.Headers like so, and add any headers we need to it.

var headers = std.http.Headers{ .allocator = allocator };
defer headers.deinit();

try headers.append("accept", "*/*");

Now that we have all of those, we can finally start a request and make a connection. If we just want to make a GET request and keep all of the default options, then the following is all we need.

var req = try client.request(.GET, uri, headers, .{});
defer req.deinit();

However, if we take a look at std.http.Client.Options (the third parameter to request), we do get some configuration options:

pub const Options = struct {
    version: http.Version = .@"HTTP/1.1",

    handle_redirects: bool = true,
    max_redirects: u32 = 3,
    header_strategy: HeaderStrategy = .{ .dynamic = 16 * 1024 },

    pub const HeaderStrategy = union(enum) {
        /// In this case, the client's Allocator will be used to store the
        /// entire HTTP header. This value is the maximum total size of
        /// HTTP headers allowed, otherwise
        /// error.HttpHeadersExceededSizeLimit is returned from read().
        dynamic: usize,
        /// This is used to store the entire HTTP header. If the HTTP
        /// header is too big to fit, `error.HttpHeadersExceededSizeLimit`
        /// is returned from read(). When this is used, `error.OutOfMemory`
        /// cannot be returned from `read()`.
        static: []u8,
    };
};

These options let us change what kind of request we're making, downgrade to HTTP/1.0 if necessary, and change how redirects are handled. But the most helpful option is likely to be header_strategy which lets us decide how the response headers are stored (either in our buffer or dynamically allocated with the client's allocator).

Getting ready to send our Request

Client.request(...) only forms a connection, it doesn't send anything. That is our job, and to do that we use Request.start(), which will send the request to the server.

If we're sending a payload (like a POST request), then we should adjust req.transfer_encoding according to our knowledge. If we know the exact length of our payload, we can use .{ .content_length = len }, otherwise use .chunked, which tells the client to send the payload in chunks.

// I'm making a GET request, so do I don't need this, but I'm sure someone will.
// req.transfer_encoding = .{ .content_length = len };

try req.start();

Now our request is in flight on its way to the server.

Pitstop: How do I send a payload?

If we're sending a POST request, it is likely we'll want to post some data to the server (surely that's why it is named that). To do that we'll need to use req.writer().

We should make sure that if we're sending data, we always finish off a request by calling req.finish(). This will send the final chunk for chunked messages (which is required) or verify that we upheld our agreement to send a certain number of bytes.

Waiting for a Response

Now that we've sent our request, we should expect the server to give us a response (assuming we've connected to a HTTP server). To do that, we use req.wait() which has quite a bit of work cut out for itself:

• So long as there is no payload attached to the request: it will handle any redirects it comes across (if we asked it to, and error if it hits the max_redirects limit)
• Read and store any headers according to our header strategy.
• Set up decompression if the server signified that it would be compressing the payload.

However, once wait() returns, the request is ready to be read.

Any response headers can be found in req.response.headers. It's a std.http.Headers so we can use getFirstValue() to read the first value of a header.

// getFirstValue returns an optional so we should can make sure that we actually got the header.
const content_type = req.response.headers.getFirstValue("content-type") orelse @panic("no content-type");

Note: any strings returned from req.response.headers may be invalidated by the last read. You should assume that they are invalidated and re-fetch them or copy them if you intend to use them after the last read.

Reading the Response

Now that we've sent our request and gotten a response, we can finally read the response. To do that we can use req.reader() to get a std.io.Reader that we can use to read the response.

For brevity's sake, I'm going to use readAllAlloc:

const body = req.reader().readAllAlloc(allocator);
defer allocator.free(body);

Finishing up

We've now completed a http request and read the response. We should make sure to call req.deinit() to clean up any resources that were allocated for the request. We can continue to make requests with the same client, or we should make sure to call client.deinit() to clean up any resources that were allocated for the client.

Complete Example

// our http client, this can make multiple requests (and is even threadsafe, although individual requests are not).
var client = std.http.Client{
    .allocator = allocator,
};

// we can `catch unreachable` here because we can guarantee that this is a valid url.
const uri = std.Uri.parse("https://example.com") catch unreachable;

// these are the headers we'll be sending to the server
var headers = std.http.Headers{ .allocator = allocator };
defer headers.deinit();

try headers.append("accept", "*/*"); // tell the server we'll accept anything

// make the connection and set up the request
var req = try client.request(.GET, uri, headers, .{});
defer req.deinit();

// I'm making a GET request, so do I don't need this, but I'm sure someone will.
// req.transfer_encoding = .chunked;

// send the request and headers to the server.
try req.start();

// try req.writer().writeAll("Hello, World!\n");
// try req.finish();

// wait for the server to send use a response
try req.wait();

// read the content-type header from the server, or default to text/plain
const content_type = req.response.headers.getFirstValue("content-type") orelse "text/plain";

// read the entire response body, but only allow it to allocate 8kb of memory
const body = req.reader().readAllAlloc(allocator, 8192) catch unreachable;
defer allocator.free(body);

How do I use a HTTP proxy?

Proxies are applied at the client level, so all you need to do is initialize the client with the proxy information.

The following will use a HTTP proxy hosted at 127.0.0.1:8080 and pass Basic dXNlcm5hbWU6cGFzc3dvcmQ= in the Proxy-Authentication header (for those who need to use an authenticated proxy).

Both port and auth are optional and will default to the default port (80 or 443) or none respectively.

var client = std.http.Client{
    .allocator = allocator,
    .proxy = .{
        .protocol = .plain,
        .host = "127.0.0.1",
        .port = 8080,
        .auth = "Basic dXNlcm5hbWU6cGFzc3dvcmQ=",
    },
};

What Next?

If it wasn't obvious, the HTTP client is still very much a work in progress. There are a lot of features that are missing, and a lot of bugs that need to be fixed. If you're interested in helping out, trying to adopt std.http into your projects is a great way to help out.