Reference

Reference material for services you can self host.

1: How email works
2: The TURN protocol

1 - How email works

Reference material about the internal workings of email systems.

Mail system architecture

Sending mail in a typical email setup looks like this:

The different components in this diagram are:

A user agent is a command line or GUI program used to compose, read, reply to and forward messages. It also stores mail locally on an end user’s computer.
A mail server is a dedicated computer used to send and receive mail over the Internet.
Message transfer agents (MTAs) perform the actual mail transfer (sending and receiving) using the Simple Mail Transfer Protocol (SMTP). Mail is always sent from an MTA client to an MTA server, with the latter running as a service and listening on ports 25 or 587 of the mail server machine.
Mailboxes (marked as Boxes in our diagram) are files or directories on the mail server which are used to store received mail. A mail server for a particular domain will typically have multiple mailboxes, with the destination mailbox for an email specified as part of the email address: <mailbox-name>@<domain-name>.
Message Access Agents (MAAs) allow a user agent program to access received mail from the mail server. These typically use the IMAP or the POP3 protocols (see below).

The steps involved in sending an email are as follows:

A user agent (UA) program (usually running on a laptop, PC, or smartphone) is used to write and initiate sending of an email.
The UA program then uses a mail transfer agent (MTA) client to transfer the outgoing mail to an MTA server via SMTP.
The MTA server queues the outgoing email in a spool.
The outgoing email is picked up by an MTA client on the mail server. This client usually runs continuously, waiting for new emails to appear on the spool.
The email is sent by the MTA client over the internet to an MTA server on the destination mail server. The destination mail server is located using the <domain-name> in an email address <mailbox-name>@<domain-name>, as well as an MX DNS record (see below).
The MTA server at the destination mail server stores the received mail in the appropriate mailbox. The mailbox name is determined from the email address: <mailbox-name>@<domain-name>.
On receiving a request from an MAA client, the MAA server at the destination mail server retrieves all received mail from the appropriate mailbox.
It then forwards it to the MAA client.
The MAA client makes newly received mail available to the UA program.

Note that SMTP is a push protocol: mail is pushed from a client to a server until it reaches the destination mailbox; while IMAP and POP3 are pull protocols: mail is requested from the mailbox of the destination mail server by a client. This architecture is well suited for the Internet, where end users typically operate machines (such as PCs and smartphones) that are intended to run client (and not server) software. This also explains the division into MTAs and MAAs within a mail system.

Simple Mail Transfer Protocol (SMTP)

IMAP/POP3

Additional links of interest:

2 - The TURN protocol

Reference material about the TURN protocol.

Network Address Translation (NAT)

When IPv4 was introduced in 1981, the Internet was far smaller than it is today. The choice to use 32 bit numbers as IP addresses was short sighted, as it allows a mere 4.3 billion devices to be addressed. By the 90s it was clear that IPv4 addresses were going to limit the growth of the Internet. At the same time, a new system could not be easily adopted without losing compatibility with the already existing Internet.

Network address translation (NAT) was proposed as a solution to this problem. Under NAT, each individual network interface is assigned a private IPv4 address which uniquely identifies the interface within its local area network (LAN). These private IPv4 addresses are taken from one of three standard ranges, depending on the required size of the LAN:

10.0.0.0/8 for the largest networks
172.16.0.0/12 for medium-sized networks
192.168.0.0/16 for small networks (home networks are often configured to use a subrange of this).

The router uses NAT to act as a gateway to the Internet. When an interface in a LAN wishes to establish a connection to a public IPv4 address on the Internet, it sends its request to the router. The router selects a public IPv4 address from a small pool assigned to it, and then stores the following details in a NAT table:

Private IPv4 address and port of the network interface that wishes to connect.
Public IPv4 address and port of the destination.
Public IPv4 address and port assigned by the router for this communication.

The router then uses the public IPv4 address it has selected to establish a connection with the destination, forwarding any data it receives from the destination to the private network interface using the information in the NAT table.

This is shown schematically in the following diagram:

Once the connection is terminated, the relevant entry in the NAT table is removed, and the public IPv4 address assigned by the router is free to use in a new connection.

In this way, a router can multiplex requests from many different private network interfaces using a small number of IPv4 addresses, reducing the demand for public IPv4 addresses.

NAT has the disadvantage that private network interfaces cannot be addressed directly on the Internet. They must initiate any TCP or UDP connection over the Internet. This is not a fatal flaw, as the Internet is compromised mostly of client devices (smartphones, PCs, laptops, etc) which initiate connections with servers to use services over the Internet (e.g. email, web apps, etc).

However, certain applications such as real-time communication require a peer to peer (p2p) connection between two clients (e.g. two browsers). The Traversal Using Relays around NAT (TURN) protocol is one method which allows two peers contained behind NAT to communicate directly with one another despite the limitations of NAT discussed above.