One grammar for every URL

A URL looks like free-form text, but it has a strict grammar, defined by 3986 as a Uniform Resource Identifier. Once you know the five top-level parts, every URL becomes easy to read, and a parser can split any of them apart with a single pattern. Take this example and keep it in mind as the parts are introduced:

https://user@host.example.com:8443/path/to/page?q=hello&lang=en#section

The five parts, in order, are the scheme, the authority, the path, the query, and the fragment. The punctuation between them is what a parser keys on: the colon after the scheme, the double slash that introduces the authority, the question mark that begins the query, and the hash that begins the fragment.

Scheme

The scheme comes first, before the colon: https. It names the protocol or interpretation for the rest of the URL, and it is case-insensitive, so HTTPS and https mean the same thing and are normally written in lowercase. Common schemes include http and https, but also ftp, ssh, mailto, and tel. Some schemes use an authority and some do not: mailto:someone@example.com has a scheme and then jumps straight to a path with no host.

Authority: userinfo, host, and port

If the URL has a double slash after the scheme, what follows up to the next slash, question mark, or hash is the authority. It breaks into three pieces. The userinfo, before an at sign, can carry a username and optionally a password: user@ here, or user:password@ when a password is present. Embedding a password this way is discouraged, because URLs are logged and shared. The host is the machine: a domain name like host.example.com, an IPv4 literal, or an IPv6 literal in square brackets such as [2001:db8::1]. The port, after a colon, is the TCP port: 8443. When the port is omitted, the client uses the scheme's default, such as 443 for https or 80 for http.

Path, query, and fragment

The path identifies the resource on the host: /path/to/page. It is a sequence of segments separated by slashes, and an inspector can list those segments individually. The query, introduced by a question mark, carries parameters: q=hello&lang=en. Its internal structure is a convention worth its own article on query strings. The fragment, introduced by a hash, points within the resource: section. The fragment is special in one important way: it is handled entirely by the client and is never sent to the server, which is why it is a good place to keep data you do not want in server logs.

Reading them apart

The value of seeing a URL split into these named parts is that subtle mistakes become obvious. A password hiding in the userinfo, a non-default port, a host that is really an IP address, a parameter buried in a long query: all of them stand out once the URL is decomposed. The URL inspector does exactly this decomposition, faithfully and without normalizing the URL, so what you see is what the URL actually says.

This series, in order

This article opens the series URLs, read exactly. With the five parts named, the series reads the rest of the grammar in order (the names for the thing, the escaping, the query, resolution, international hosts, where a domain is registered) and ends with the tricks that rely on a reader not reading that exactly.

  1. URI, URL, URN: What's Actually the Difference?. Identify, locate, name: what the three acronyms mean in a specification and in conversation.
  2. Percent-encoding (URL encoding), explained. Why a URL writes some characters as %XX, which ones are safe as they are, and how it differs from .
  3. Query Strings: Parameters, Plus Signs, and Repeated Keys. Separators, the plus sign that sometimes means a space, repeated keys, and why no single standard governs them.
  4. Relative URLs and How They Resolve. How a browser completes a relative link against its base, and what ./, ../ and a leading slash do.
  5. URL Encoding and Internationalized Hosts. Percent-encoding for paths and queries, punycode for host names, and why internationalised hosts serve phishing.
  6. Public suffixes and the registered domain (eTLD+1). The public suffix and the registered domain (eTLD+1), why the last two labels are the wrong answer, and where it decides cookies and certificates.
  7. Deceptive URLs: Reading Past the Tricks. The userinfo trick, redirect parameters and lookalike characters, and the one habit that finds the real host.
  8. IP Address Obfuscation Tricks. One address written in decimal, octal, hex or IPv4-mapped IPv6, and why a filter must decode rather than match strings.