(std http)

Reference

Overview

HTTP requests: a request is a value you build, then send blocking, suspending or as an event.

(import (std http))

(defun (main args)
  (match (fetch (http-get "https://example.com/fruit"))
    ((Err e) 1)
    ((Ok r)
     (println (response-status r))
     (println (response-text r))
     (response-close r)
     0)))
;; Streaming a body a line at a time, without holding it in memory.
;; fetch, port-eof? and read-line all suspend because this is a defbjo;
;; the same source blocks in a defun.
(defbjo (count-lines url)
  (match (fetch (http-get url))
    ((Err e) 0)
    ((Ok r)
     (let ((port (response-port r)))
       (let count ((n 0))
         (if (port-eof? port)
             (let () (response-close r) n)
             (let () (ignore (read-line port)) (count (+ n 1)))))))))
;; JSON is two modules rather than one: this one has no serializer.
(import (text json))

(defbjo (fetch-json url)
  (match (fetch (http-get url #:headers [("Accept" . "application/json")]))
    ((Err e) (Err "the request failed"))
    ((Ok r)
     (let ((parsed (read-json (response-port r))))
       (response-close r)
       parsed))))

A request is a value you build and then send. How it is sent, blocking, suspending, or as an event, is the only thing that differs between fetch and fetching.

There is no C# in this module: System.Net.Http is reached with import/extern and the #:async machinery, and the response body is an ordinary InputPort.

Two things are deliberately not what .NET does. A status code is a value, never an exception: 404 is an Ok holding a 404. And HttpClient.Timeout is switched off, because a deadline belongs to the event system (see below).

The ambient cancellation token reaches .NET by itself, so with-cancel and with-deadline stop a request in flight without anything being passed along.

Do not drain a body with port->seq. A seq body cannot hold an await, so its reads park the fiber’s thread, and the compiler says so. port->vec, port->chan and a loop like the one in the example are the suspending ways to drain a port.

What to watch out for.

See also: http-get, http-post, fetch, fetching, response-close, with-response

Reference

Types

Functions

Values

Macros

Types

Method

union ↑ Types

An HTTP method: GET, POST, PUT, PATCH, DELETE or HEAD.

See also: request

Body

union ↑ Types

What a request carries: nothing, text, JSON text, a form, or a body of your own.

Empty
No body.
Text
Plain text, sent as text/plain; charset=utf-8.
JsonText
Text that is already serialized JSON, sent as application/json; charset=utf-8.
Form
Fields, percent-encoded and sent as application/x-www-form-urlencoded.
Other
A Custom body: your own content type and text.

JsonText takes text that is already serialized; this module has no JSON writer. It is spelled that way rather than Json because (text json) has a type called Json.

Every body is sent as UTF-8.

See also: http-post, request, Custom

Custom

record ↑ Types

A body with a content type of your own.

content-type
The Content-Type header to send, as written.
text
The body, sent as UTF-8.
(request PUT "https://example.com/note"
         #:body (Other (Custom (content-type "text/csv") (text "a,b\n1,2"))))

See also: Body

Pair

type alias ↑ Types

A name and a value: a header, a query parameter or a form field.

("Accept" . "application/json")

Request

record ↑ Types

A request, not yet sent.

method
The HTTP method.
url
The url, which must be http or https. It may already carry a query.
query
Query parameters, percent-encoded and appended to the url.
headers
Headers, for the request and for its body alike.
body
The body, or Empty.

See also: request, http-get, http-post, fetch

Response

record ↑ Types

An HTTP response: status code, headers, and an unread body stream.

Opaque: the accessors are the whole of it. The CLR reply inside owns a live connection, so a response has to be closed, with response-close, ignore or with-response.

See also: response-status, response-text, response-port, response-close, with-response

Client

type alias ↑ Types

A System.Net.Http.HttpClient: connections and their settings.

See also: make-client, current-http-client

Functions

request

function ↑ Functions

(: request (-> Method string (#:query (Vec Pair)) (#:headers (Vec Pair)) (#:body Body) Request))
(request method url #:query (vec-empty) #:headers (vec-empty) #:body Empty)

A request with any method.

method
The method.
url
Where to send it.
#:query
Query parameters, appended to the url.
#:headers
Headers, for the request or its body.
#:body
The body.
returns
The request, not yet sent.
(request PUT "https://example.com/note"
         #:body (Other (Custom (content-type "text/csv") (text "a,b\n1,2"))))

Query parameters are percent-encoded and appended; a url that already has a query gets & rather than ?. They go before a #fragment. Each name and value is encoded by encode-query-part, and a form body by encode-form, from (std url).

A header goes on the request or on the body depending on which it is, Accept on the one and Content-Type on the other, and both are written in the same vector. Naming a body header on a request with no body is an Err when it is sent, as is a url that does not parse or is not http or https.

See also: http-get, http-post, fetch

http-get

function ↑ Functions

(: http-get (-> string (#:query (Vec Pair)) (#:headers (Vec Pair)) Request))
(http-get url #:query (vec-empty) #:headers (vec-empty))

A GET request.

url
Where to send it.
#:query
Query parameters, appended to the url.
#:headers
Headers.
returns
The request, not yet sent.
(http-get "https://example.com/search"
     #:query   [("q" . "banana split") ("limit" . "10")]
     #:headers [("Accept" . "application/json")])

See also: http-post, request

http-post

function ↑ Functions

(: http-post (-> string Body (#:query (Vec Pair)) (#:headers (Vec Pair)) Request))
(http-post url body #:query (vec-empty) #:headers (vec-empty))

A POST request with a body.

url
Where to send it.
body
The body.
#:query
Query parameters, appended to the url.
#:headers
Headers, for the request or its body.
returns
The request, not yet sent.
(http-post "https://example.com/basket"
      (JsonText "{\"fruit\":\"banana\"}")
      #:headers [("Authorization" . "Bearer hunter2")])

(http-post "https://example.com/login" (Form [("user" . "bjoli") ("pass" . "banan")]))

See also: http-get, request, Body

fetch

function ↑ Functions

(: fetch (-> Request (Result Exception Response)))
(fetch req)

Sends a request and waits for response headers.

req
The request.
returns
Ok with the response, whatever its status, or Err if it could not be built or sent.
;; A deadline over a whole request, including the body. A bjoroutine,
;; because a scope waits for its children before it returns, and waiting
;; is a yield point.
(defbjo (main args)
  (with-deadline (seconds 5)
    (match (fetch (http-get "https://example.com/slow"))
      ((Ok r) (println (response-text r)) (response-close r) 0)
      ((Err e) (println "gave up") 1))))

Colourless: inside a bjoroutine it suspends and the thread goes back to the pool, outside one it parks the calling thread. There is no second name, and a function that calls it needs no colour of its own.

Failure is a value. A status code is not a failure: 404 is an Ok holding a 404. The response is closed by whoever receives it.

See also: fetching, response-close, with-response

fetching

function ↑ Functions

(: fetching (-> Request (Event (Result Exception Response))))
(fetching req)

A request as an event: sent when the branch is reached, cancelled if it loses.

req
The request.
returns
An event evaluating to the result that fetch would return.
;; Two mirrors, whichever responds first. The loser's request is
;; cancelled, not merely ignored.
(type (: Outcome (Union (: Got (Result Exception Response)) TimedOut)))

(defbjo (from-a-mirror)
  (match (sync (choose (wrap (fetching (http-get "https://eu.example.com/fruit")) (fun (a) (Got a)))
                       (wrap (fetching (http-get "https://us.example.com/fruit")) (fun (a) (Got a)))
                       (wrap (timeout (seconds 2)) (fun (fired) TimedOut))))
    ((Got (Ok r))  (let ((body (response-text r))) (response-close r) body))
    ((Got (Err e)) "failed")
    (TimedOut      "no mirror responded")))
;; Retrying. A fresh request is built at every sync, so a retry loop is
;; just a loop.
(defbjo (with-retries req tries)
  (match (sync (fetching req))
    ((Ok r) (Ok r))
    ((Err e) (if (= tries 0)
                 (Err e)
                 (let () (sync (timeout (milliseconds 250))) (with-retries req (- tries 1)))))))

What to reach for when the request has to be a branch of a choose. Nothing is sent until the branch is reached.

The message is built afresh at every sync: a .NET request message may be sent only once, and an event may be synced any number of times, so the same event can be retried.

A branch that loses after the headers have arrived leaks that response, since nothing is left to close it.

See also: fetch

response-status

function ↑ Functions

(: response-status (-> Response int))
(response-status r)

The status code.

r
The response.
returns
The code, such as 200 or 404.

See also: response-ok?, response-reason

response-ok?

function ↑ Functions

(: response-ok? (-> Response bool))
(response-ok? r)

Whether the status is 2xx.

r
The response.
returns
#t for a status from 200 to 299.

Nothing here throws on a 4xx or 5xx: which codes are failures is the caller’s business.

See also: response-status

response-reason

function ↑ Functions

(: response-reason (-> Response string))
(response-reason r)

The reason phrase, such as “Not Found”.

r
The response.
returns
The phrase, or the empty string over HTTP/2, which has none. Do not parse it.

See also: response-status

response-header

function ↑ Functions

(: response-header (-> Response string (Option string)))
(response-header r name)

Retrieves a header value from a response.

r
The response.
name
The header's name, in any case.
returns
Its value, or None. Repeated headers are folded with ", ".
(response-header r "Content-Type")   ; the same as "content-type"

See also: response-headers

response-headers

function ↑ Functions

(: response-headers (-> Response (Map string string)))
(response-headers r)

All headers from a response, including content headers.

r
The response.
returns
A map from lower-cased names to values. Repeated headers are folded with ", ".

Folding is wrong for Set-Cookie alone: folded cookies cannot be taken apart again.

See also: response-header

response-text

function ↑ Functions

(: response-text (-> Response string))
(response-text r)

The whole body, as text.

r
The response.
returns
The body, decoded by the charset the server named, or UTF-8.
raises System.Net.Http.HttpRequestException
When the charset the server named is not an encoding (text encodings) knows.

Reads it through response-port, so it may be called once, and not after response-port. Colourless: reading suspends inside a bjoroutine and blocks outside one, exactly as reading a file does.

See also: response-port, response-bytes-port

response-port

function ↑ Functions

(: response-port (-> Response InputPort))
(response-port r)

The body as a port, to read as it arrives.

r
The response.
returns
A InputPort over the body.
raises System.Net.Http.HttpRequestException
When the charset the server named is not an encoding (text encodings) knows.

So read-line, port->chan and read-json all work on it. Reading suspends inside a bjoroutine and blocks outside one, exactly as reading a file does.

Once per response, and once between this and response-bytes-port: a second call hands out a second reader over a stream the first one is already draining.

The text is decoded by the charset the Content-Type names, by any name or alias encoding-named knows, and UTF-8 is assumed when it names none. A charset nothing knows raises, since a body decoded by a guess is wrong without saying so; response-bytes-port still reads it.

The declared charset is the one used, and a byte order mark is not treated specially. A UTF-8 body is the port’s bytes as they arrived; a body in another charset is transcoded to UTF-8 on the way in, as reencode-input-port does. Bytes that do not decode read as U+FFFD, as on any port.

See also: response-text, response-bytes-port

response-bytes-port

function ↑ Functions

(: response-bytes-port (-> Response InputPort))
(response-bytes-port r)

The body’s bytes as they arrived, whatever the charset says.

r
The response.
returns
A InputPort over the body, undecoded.
;; A body in a charset the server names wrongly, decoded by the one it is.
(def text (reencode-input-port (response-bytes-port r) windows-1252))
(read-all text)

As Racket’s HTTP client hands a body over. For one that is not text, an image or a zip, read with read-some or read-bytes; for text in a charset to choose yourself, reencode-input-port or bytes->string. Once per response, and not as well as response-port.

See also: response-port, bytes->string

response-close

function ↑ Functions

(: response-close (-> Response void))
(response-close r)

Releases the response’s connection.

r
The response.

A response holds a connection until it is closed. (ignore r) does the same, so a response dropped in statement position releases it rather than leaking it.

See also: with-response

make-client

function ↑ Functions

(: make-client (-> (#:connection-seconds double) (#:redirects bool) Client))
(make-client #:connection-seconds 120.0 #:redirects #t)

A new client, with settings of its own.

#:connection-seconds
How long a pooled connection lives before it is replaced.
#:redirects
Whether redirects are followed.
returns
The client. Responses are decompressed, and it has no timeout.

A pooled connection is recycled after connection-seconds: one that lived forever would outlast a DNS change and keep talking to an address that stopped being right.

HttpClient.Timeout is off on purpose. It applies to every call made through the client, cannot be varied per request, and arrives as a cancellation that cannot be told from a real one. Race a (timeout d) branch, or use with-deadline.

See also: current-http-client

Values

current-http-client

value ↑ Values

(: current-http-client (Param Client))

The client every send uses.

(parameterize ((current-http-client (make-client #:redirects #f)))
  (fetch (http-get "https://example.com/redirects-somewhere")))

A (Param Client). Parameterize another one in for a proxy, a test double or a second set of connection settings.

See also: make-client

Macros

with-response

macro ↑ Macros

(with-response (name form) body ...)

Runs body with a response, and closes it on the way out.

name
Bound to the (Result Exception Response).
form
What produces it, such as (fetch req).
body ...
What to run. At least one form.
(with-response (r (fetch (http-get "https://example.com/fruit")))
  (match r
    ((Ok got) (response-text got))
    ((Err e) "")))

The response is closed however the body leaves, by returning or by raising. An Err has nothing to close.

See also: fetch, response-close