(std url)

Reference

Overview

URLs: read, written, resolved and percent-encoded, by RFC 3986.

(import (std url))

;; Write one. Each value is encoded for the part that it is in.
(link '(url https "example.com" (path "users" ,name)
            (query (page 2) (tag ,tag))
            (fragment "top")))
;; => "https://example.com/users/Ada%20L?page=2&tag=a%26b#top"

(link '(path "users" ,id "edit"))       ;; => "/users/42/edit"
;; Read one, and find a link on its page.
(match (parse-url "https://example.com/a/b?q=1")
  ((Err why) why)
  ((Ok page)
   (def (Ok ref) (parse-url "../c") :propagate)
   (url->string (url-resolve page ref))))   ;; => "https://example.com/c"

Writing a URL. A URL is written as a quoted literal, and the shape of the literal says which part each value is in. A value in (path ...) is one segment, so a / in it is encoded and does not make a second segment. A value in (query ...) is a key or a value, so a & or a = in it is encoded and does not make a second parameter. You do not choose an encoder, so you cannot choose the wrong one.

'(url https "example.com" ...)a URL with a scheme and a host
'(path "a" ,b ...)a path on this site: /a/b
(port 8080)in url: the port
(user "ada")in url: the user before @
(path seg ...)in url: the path
(query (key value) (key) ...)the query; in url or in path
(fragment "x")the fragment; in url or in path

In a (url ...), the parts after the host come in any order, each at most once. A (path ...) in a (url ...) has only segments. In a (path ...) on this site, the query and the fragment come after the segments.

A segment, a key and a value is a string, a symbol or an int, written or computed with ,. A computed (Vec string) is several: several segments in a path, or one key with several values in a query, tag=a&tag=b. A key, a user and a fragment are one text. A computed pair is ,(Tuple key value), and many pairs are ,@(list-map ParamPair pairs).

Link is a grammar (see def/grammar). The compiler reads the literal where you write it. (qeury ...), (port 1 2), a part given two times, a parameter with more than one value, and a (url ...) without a scheme or a host are compile errors.

The scheme, the host and the port are checked parts. The compiler runs scheme, host and port on a literal one, and refuses a scheme that is not one, a host that is not one, and a port that is not from 0 to 65535. A host that is not ASCII is written as IDNA writes it, and an IPv6 address gets its brackets. A computed scheme, host or port must be a Scheme, a Host or a Port. Call the check to make one, and handle the Err:

(match (host text) ((Ok h) (link '(url https ,h))) ((Err why) why))

Thus make-url and link do not fail.

A number in a literal has a type only when it is written there or has one already. (def id 42) has none yet, and the literal refuses ,id. Write (def (: id int) 42), or ,(int->string n).

Reading a URL. parse-url reads RFC 3986 strictly: a URL, or a relative reference such as ../a?b. A space, a character that is not ASCII and a % without two hex digits are refused, with the column. Such text has to be percent-encoded before it is a URL. A browser accepts more and corrects it (the WHATWG URL standard). This module does not.

A Url keeps each part as it was written, percent-encoding included, so url->string gives back the same text. There is one exception: an empty port, http://h:/, is written without its :, as RFC 3986 6.2.3 recommends. The scheme and the host keep their case. The decoded forms are functions: url-segments and url-params.

Not here. Normalisation beyond remove-dot-segments, comparison of URLs, IRIs, and a zone in an IPv6 address (fe80::1%25eth0), which this module refuses.

See also: link, parse-url, url-resolve, url-params, encode-segment

Reference

Types

Functions

Types

Url

record ↑ Types

A URL, or a relative reference, with each part as it was written.

scheme
The scheme without its :, such as "https"; None for a relative reference.
authority
The user info, the host and the port; None when there is no //.
path
The path, percent-encoded. It can be empty.
query
The query without its ?, percent-encoded; None when there is no ?.
fragment
The fragment without its #, percent-encoded; None when there is no #.

None is a part that is not there. That is not the same as a part that is empty: http://h/? has the query "".

See also: parse-url, url->string, make-url

Authority

record ↑ Types

The part of a URL after //: the user info, the host and the port.

userinfo
The user info, percent-encoded, without the at sign after it; None when there is none.
host
The host as written. An IP literal keeps its brackets: [::1].
port
The port; None when there is none.

See also: url-authority, url-host, url-port

union ↑ Types

A URL written as a quoted literal: '(url ...) or '(path ...).

LinkUrl
(url scheme host part ...): a URL with a scheme and a host.
LinkPath
(path segment ... (query ...) (fragment ...)): a path on this site.

Link is a grammar (see def/grammar). The module’s documentation has the parts. The export of Link also exports UrlParts, PathParts, QueryParam, UrlText and UrlPiece.

See also: link, make-url, UrlParts, PathParts, Scheme, Host, Port

UrlParts

record ↑ Types

The parts of a (url ...) after the host. Each part is there at most once, in any order.

port
(port 8080).
user
(user "ada").
path
(path segment ...).
query
(query (key value) ...).
fragment
(fragment "top").

See also: Link

PathParts

record ↑ Types

The parts of a (path ...): the segments, then the query and the fragment.

segments
The segments.
query
(query (key value) ...).
fragment
(fragment "top").

See also: Link

QueryParam

union ↑ Types

One parameter of a (query ...).

ParamForm
(key value), or (key) with no value.
ParamPair
A computed pair: ,(Tuple key value).

See also: Link, params->query

UrlText

union ↑ Types

A key, a user or a fragment in a literal: one text.

TextStr
A string.
TextSym
A symbol, as its name.
TextInt
An int.

See also: Link, UrlPiece

UrlPiece

union ↑ Types

A segment or a query value in a literal: one text, or several.

PieceOne
One text.
PieceMany
A computed (Vec string): several segments, or one key with several values.

See also: Link, UrlText

Scheme

record ↑ Types

The scheme of a URL literal, such as https, that scheme accepted.

See also: scheme, scheme->string, Link

Host

record ↑ Types

The host of a URL literal, that host accepted, in the form a URL writes it.

See also: host, host->string, Link

Port

record ↑ Types

The port of a URL literal, from 0 to 65535, that port accepted.

See also: port, port->int, Link

Functions

url-scheme

function ↑ Functions

(: url-scheme (-> Url (Option string)))
(url-scheme u)

The scheme of a URL, without its :.

u
The URL.
returns
(Some "https"), or None for a relative reference.

url-authority

function ↑ Functions

(: url-authority (-> Url (Option Authority)))
(url-authority u)

The authority of a URL: user info, host and port.

u
The URL.
returns
The Authority, or None when the URL has no //.

url-path

function ↑ Functions

(: url-path (-> Url string))
(url-path u)

The path of a URL, percent-encoded as written.

u
The URL.
returns
The path. It can be empty.

See also: url-segments

url-query

function ↑ Functions

(: url-query (-> Url (Option string)))
(url-query u)

The query of a URL, percent-encoded, without its ?.

u
The URL.
returns
The query, or None when there is no ?.

See also: url-params

url-fragment

function ↑ Functions

(: url-fragment (-> Url (Option string)))
(url-fragment u)

The fragment of a URL, percent-encoded, without its #.

u
The URL.
returns
The fragment, or None when there is no #.

See also: percent-decode

authority-userinfo

function ↑ Functions

(: authority-userinfo (-> Authority (Option string)))
(authority-userinfo a)

The user info of an authority, percent-encoded.

a
The authority.
returns
The user info without the at sign after it, or None.

authority-host

function ↑ Functions

(: authority-host (-> Authority string))
(authority-host a)

The host of an authority, as written.

a
The authority.
returns
The host. An IP literal keeps its brackets.

authority-port

function ↑ Functions

(: authority-port (-> Authority (Option int)))
(authority-port a)

The port of an authority.

a
The authority.
returns
The port, or None when there is none.

url-host

function ↑ Functions

(: url-host (-> Url (Option string)))
(url-host u)

The host of a URL, as written.

u
The URL.
returns
The host, or None when the URL has no authority.

url-port

function ↑ Functions

(: url-port (-> Url (Option int)))
(url-port u)

The port of a URL.

u
The URL.
returns
The port, or None when the URL has no port written.

A scheme’s default port is not filled in: https://h/ has no port, not 443.

url-absolute?

function ↑ Functions

(: url-absolute? (-> Url bool))
(url-absolute? u)

Whether a URL has a scheme.

u
The URL.
returns
#t for https://h/ and urn:isbn:0451450523; #f for /a and //h/a.

parse-url

function ↑ Functions

(: parse-url (-> string (Result string Url)))
(parse-url s)

Reads a URL or a relative reference, by RFC 3986.

s
The text.
returns
The Url, or why it is not one, with the column.
(parse-url "https://ada@example.com:8443/a/b%2Fc?x=1#top")
(parse-url "../a?b")              ;; a relative reference
(parse-url "http://h/a b")
;; => (Err "column 11: a space is not allowed in the path, and has to be percent-encoded")

Refused: a byte that the part does not allow (a space, a character that is not ASCII, a control character), a % without two hex digits, a port that is not digits or is more than 65535, a [ with no ], an IP literal that is not an IPv6 address or an IPvFuture address, and a relative reference with a : in its first segment. a:b is the scheme a, so RFC 3986 writes that reference ./a:b.

See also: url->string, url-resolve

url->string

function ↑ Functions

(: url->string (-> Url string))
(url->string u)

Writes a URL as text.

u
The URL.
returns
The text. For a URL from parse-url it is the text that was read, except an empty port.

See also: parse-url, link

url-resolve

function ↑ Functions

(: url-resolve (-> Url Url Url))
(url-resolve base ref)

Reads a reference where a base URL is, as a browser reads a link on a page.

base
The URL of the page. It should have a scheme.
ref
The reference, such as ../a, ?q or //h/x.
returns
The URL that ref points to.
;; base: http://a/b/c/d;p?q
;; "g"        => http://a/b/c/g
;; "../g"     => http://a/b/g
;; "?y"       => http://a/b/c/d;p?y
;; "//g"      => http://g

The algorithm of RFC 3986 5.2.2. A reference with a scheme is a URL of its own, and only its dot segments are removed.

See also: remove-dot-segments, parse-url

remove-dot-segments

function ↑ Functions

(: remove-dot-segments (-> string string))
(remove-dot-segments path)

Takes the . and .. segments out of a path.

path
A path, percent-encoded.
returns
The path without dot segments.
(remove-dot-segments "/a/b/../c/./d")   ;; => "/a/c/d"
(remove-dot-segments "/a/..")           ;; => "/"

RFC 3986 5.2.4. A .. above the root is dropped, so /a/../../b is /b. A path that ends in . or .. ends in /.

url-segments

function ↑ Functions

(: url-segments (-> Url (Result string (List string))))
(url-segments u)

The segments of a URL’s path, decoded.

u
The URL.
returns
The segments, or why one does not decode.
;; /a/b%2Fc/  =>  (Ok ("a" "b/c" ""))

The / at the start of the path begins the first segment. A / at the end leaves an empty last segment. An encoded / is part of its segment, which is why the segments are decoded one by one and not the path as one string. The empty path has no segments.

url-params

function ↑ Functions

(: url-params (-> Url (Result string (List (Tuple string string)))))
(url-params u)

The parameters of a URL’s query, decoded, in order.

u
The URL.
returns
The pairs, as query->params reads them; no pairs when there is no query.

See also: query->params, param-ref, param-ref-all

query->params

function ↑ Functions

(: query->params (-> string (Result string (List (Tuple string string)))))
(query->params q)

Reads a query as key and value pairs, in order.

q
The query, without its ?.
returns
The pairs, decoded, or why one does not decode.
(query->params "a=1&b=two+words&b=%C3%A4&flag")
;; => (Ok (list (Tuple "a" "1") (Tuple "b" "two words") (Tuple "b" "ä") (Tuple "flag" "")))

A + is a space, as a form writes it. A parameter with no = has the value "". An empty parameter, a=1&&b=2, is no parameter. A key that is there two times is in the list two times, so param-ref-all can find every value.

params->query

function ↑ Functions

(: params->query (-> (List (Tuple string string)) string))
(params->query params)

Writes key and value pairs as a query.

params
The pairs, in order.
returns
The query, without the ?.
(params->query (list (Tuple "q" "a&b") (Tuple "page" "2")))   ;; => "q=a%26b&page=2"

See also: encode-query-part, link

param-ref

function ↑ Functions

(: param-ref (-> (List (Tuple string string)) string (Option string)))
(param-ref params key)

The first value of a key.

params
The pairs.
key
The key.
returns
(Some value), or None when the key is not there.

param-ref-all

function ↑ Functions

(: param-ref-all (-> (List (Tuple string string)) string (List string)))
(param-ref-all params key)

Every value of a key, in order.

params
The pairs.
key
The key.
returns
The values; empty when the key is not there.

encode-segment

function ↑ Functions

(: encode-segment (-> string string))
(encode-segment s)

Percent-encodes text as one segment of a path.

s
The text.
returns
The segment. A /, ? and # are encoded.
(encode-segment "a/b c")   ;; => "a%2Fb%20c"

What RFC 3986 lets a segment hold stays as it is: letters, digits, - . _ ~, ! $ & ' ( ) * + , ; =, : and @. Every other byte of the UTF-8 is %XX.

See also: percent-decode, link

encode-query-part

function ↑ Functions

(: encode-query-part (-> string string))
(encode-query-part s)

Percent-encodes text as a key or a value of a query.

s
The text.
returns
The key or the value. A &, =, +, ; and # are encoded, and a space is %20.

A + is encoded because a server reads it as a space. A space is %20, which every server reads as a space. So the value comes back the same, whichever way the server reads a query.

See also: params->query, encode-form

encode-fragment

function ↑ Functions

(: encode-fragment (-> string string))
(encode-fragment s)

Percent-encodes text as a fragment.

s
The text.
returns
The fragment. A # and a space are encoded.

encode-userinfo

function ↑ Functions

(: encode-userinfo (-> string string))
(encode-userinfo s)

Percent-encodes text as a user name or a password.

s
The text.
returns
The text, with : and the at sign encoded.

encode-form

function ↑ Functions

(: encode-form (-> string string))
(encode-form s)

Encodes text as application/x-www-form-urlencoded writes it.

s
The text.
returns
The text: letters, digits and * - . _ as they are, a space as +, every other byte %XX.

This is what a browser sends for a form. For a link, use encode-query-part or a literal.

See also: form-decode, encode-query-part

percent-decode

function ↑ Functions

(: percent-decode (-> string (Result string string)))
(percent-decode s)

Decodes percent-encoded text.

s
The text.
returns
The decoded text, or why it does not decode: a % without two hex digits, or bytes that are not UTF-8.

A + stays a +. Use form-decode for a form, where it is a space.

See also: form-decode

form-decode

function ↑ Functions

(: form-decode (-> string (Result string string)))
(form-decode s)

Decodes text that a form encoded: + is a space.

s
The text.
returns
The decoded text, or why it does not decode.

See also: percent-decode, encode-form

scheme

function ↑ Functions

(: scheme (-> Symbol (Result string Scheme)))
(scheme s)

Checks a scheme: a letter, then letters, digits, +, - and ..

s
The scheme, a symbol such as 'https.
returns
The Scheme, or why it is not one.
(match (scheme (string->symbol text))
  ((Ok s) (link '(url ,s "example.com")))
  ((Err why) why))

The compiler runs this check on a literal scheme in a (url ...). A computed scheme must be a Scheme, so call this function to make one.

See also: Scheme, Link

host

function ↑ Functions

(: host (-> string (Result string Host)))
(host h)

Checks a host, and gives the form a URL writes it in.

h
The host: a name, an IPv4 address, or an IPv6 address with or without brackets.
returns
The Host, or why it is not one.
(host "bücher.example")   ;; => (Ok ...), written xn--bcher-kva.example
(host "::1")              ;; => (Ok ...), written [::1]

An IPv6 address gets its brackets. A name that is not ASCII becomes its IDNA form. The compiler runs this check on a literal host in a (url ...). A computed host must be a Host, so call this function to make one.

See also: Host, Link

port

function ↑ Functions

(: port (-> int (Result string Port)))
(port n)

Checks a port: from 0 to 65535.

n
The port.
returns
The Port, or why it is not one.

The compiler runs this check on a literal port in a (port ...). A computed port must be a Port, so call this function to make one.

See also: Port, Link

scheme->string

function ↑ Functions

(: scheme->string (-> Scheme string))
(scheme->string s)

The text of a scheme.

s
The scheme.
returns
The scheme as text, without its :.

See also: scheme

host->string

function ↑ Functions

(: host->string (-> Host string))
(host->string h)

The text of a host, as a URL writes it.

h
The host.
returns
The host as text.

See also: host

port->int

function ↑ Functions

(: port->int (-> Port int))
(port->int p)

The number of a port.

p
The port.
returns
The port as an int.

See also: port

make-url

function ↑ Functions

(: make-url (-> Link Url))
(make-url l)

The Url that a literal stands for.

l
The literal.
returns
The URL, with each value encoded for its part.

This function does not fail. The grammar refuses a bad literal at compile time, and the checks refuse a bad scheme, host or port.

See also: link, Link

function ↑ Functions

(: link (-> Link string))
(link l)

The text of a URL written as a literal.

l
The literal.
returns
(url->string (make-url l)).
(link '(path "search" (query (q ,text))))   ;; => "/search?q=a%20%26%20b"

See also: make-url, Link