How Email Works

 Computing, Email, Tutorials, Uncategorized  No Responses »
Jul 132019
 

This question popped on Twitter just now; I’m not sure the question was a serious one, but as it happens I can answer the question as if it were serious. I’ve been running relatively small mail servers for nearly 30 years, so do know a litle bit about it.

This of course does not cover lots of “bolt ons” to email such as SPF, DKIM, DMARC, and that’s just the simple stuff.

To keep things relatively sane, I’m going to gloss over an awful lot of details – for example, there are all sorts of ways of composing and sending emails, but I will assume you’re using some sort of web interface such as Google Mail.

So you compose an email with an interesting picture or two, and click “Send”.

Hang on a bit! You also did some technical stuff when you entered an email address into the “To” field (if you just enter someone’s name, something somewhere is changing that to an email address) and possibly something summarising the email (or trying to get attention) into the “Subject” field.

Headers

Those two headers you have filled in – ‘To’ and ‘Subject’ – are just two of a whole collection of headers that are usually invisible. They are invisible because your mail client is hiding them from you; for good reasons as they’re almost always an unnecessary distraction.

But when you have to diagnose an email problem, seeing the full set of raw headers is kind of essential even if it is dead simple (for the relatively technically sophisticated) to forge headers.

For example, if you set up a dedicated mail client rather than use your mail provider’s web-based mail client, you will have to fill out your own email address (to go into the “From” field). You can fill anything you like into that field although you are unlikely to get a reply

Email Addresses

If you fiddle with your mail client enough, you will realise that the “To” field can be changed to a “Cc” field or a “BCc” field. All three take email addresses, and can be used to send an email to someone but they operate in very slightly different ways.

The “To” header is for the main audience of a message.

The “Cc” header is for an audience who should also see the message, but perhaps don’t need to take any action based on the email.

The “Bcc” header is a bit special because it isn’t added to the raw mail message as a header, but is instead used to compose the envelope listing who should receive the email. If you need (or should) keep the list of people receiving an email secret, or just want to avoid an inconveniently long header for people to read then use the “Bcc” header.

Although your mail client may hide the details, email addresses look like :-

  1. some-name@example.com (a “pure” email address)
  2. “Some Name <some-name@example.com” (as normally formatted)

The Message Submission Server

So you click on “send”. What happens then?

The first thing that happens is that your mail client converts your email (attachments, rich formatting, etc.) into plain text which is why a file attachment just under your ISP’s limit on mail sizes can exceed that maximum.

To the top of that plain text version of your message, your mail client attaches some headers (again in plain text). In addition your mail client will prepare an “invisible” envelope independent of the headers – it contains just the address(es) you want to send to, and your own address.

Finally the mail client is ready to talk to the “mail submission server”. It will login (usually) with your credentials, and sends the message to the server using the “Simple Mail Transport Protocol” (SMTP).

The mail server adds your message to a queue and before it says to your mail client “Okay, I have it”, it will make sure that the message is safe on disk. Mail servers go to a lot of effort to make sure your messages don’t get “dropped on the floor” even when the operating system they are running on crashes.

The Journey

After you have submitted a mail message to the message submission server, the message is held in a queue. This is significant because email is a store and forward messaging solution, so every server your message journeys through will make sure it is stored on disk before saying to the server sending the message “Yep. Got it.”.

In theory this should mean that messages never get lost “in the system” (although practice is often different to theory). The store and forward style of architecture suits the era that email is from – back in the 1970s when email could (and did) traverse many different kinds of network rather than “the Internet” that we have today.

What does that mean today? Amongst other things, it means that whilst you may think that your message only visits two mail servers – the one belonging to your ISP and the one belonging to the recipient’s ISP – and that is certainly a valid architecture for very small organisations.

But it does not work so well for larger organisations that may handle millions, or hundreds of millions of message a day. Summing up a large architecture without actually seeing an architecture diagram for every single ISP and large organisation on the Internet is going to be a bit hit and miss, but I’ll give it a go.

Your message submission server will probably simply hand your message off to a server responsible for sending email messages across the Internet. If it cannot send to the recipient’s receiving mail server on the first run through the queue, it may well just hand the message off to a “slow message server” (processing large queues is a bit of a drag).

In either case, the recipient’s server will accept the message and then hand it over to a “mailbox server” that the recipient will connect to.

And the journey is over. At every stage, reliability is favoured over speed.

How do servers send your message to each other? Using that SMTP standard I mentioned earlier.

Speed

The speed of email is a good topic to cover here. Speed is not what email is about in the same way that the postal service for physical letters is not about speed. It’s about reliability

Your messages may be readable by the recipient within 5 minutes 99 times out of 100, but that 1 time may take a day or two. That’s just the way that email works – if you want instant messaging, use an instant messaging client.

Security

At no point have I mentioned encryption, and that is intentional. At no point is encryption required; email was designed in an era when computers were far slower and encryption was expensive (even though it was far weaker).

At every point, servers are trusting that the senders are not lying and the original sender is trusting that nobody is snooping on their emails.

To correct that there are a whole bunch of optional add-ons to the email standard to deal with that. But that is a story for another day.

End of the Pier

The Basics Of Vi

 Computing, Linux, Tutorials  No Responses »
Dec 112016
 

Vi (or vim) is one of those editors that at first appearance appears to be insanely user-hostile, and some will say it looks the same way on a second, and third glance. Yet it remains one of the most popular editors under Linux, and even if you choose another editor as your mainstay, you are likely to encounter it in use a fair bit.

(Apologies for the little line at the top of that screenshot; lazy editing)

The strange thing about vi is that nobody uses the real thing any more (well almost nobody), but instead clones of one kind or another. That is mostly because vi originated in the commercial world of Unix, and clones were written to be open-source. It is perhaps worth remembering that vi has remained the mainstay of editing under Unix-based operating systems for three decades despite there being many alternatives.

It may look plain, but vi has almost every editor feature you can think of and almost certainly a few you never thought of.

The Modes

Almost every every other editor currently in use is a modeless editor, or at least mostly modeless. Vi is different in that it has three different modes that operate differently – insert mode, command mode, and ex-mode (essentially for extended commands). Of the three modes, the insert mode is the most like other editors, although are relatively few commands to use. As you can see in the screenshot above, the words “- INSERT -” appears at the bottom of the screen whenever you are insert mode (and the same for replace mode which is effectively the same).

Most commands are performed in the command mode, which can be thought of as the default mode – there is nothing saying that you are in command mode. If in doubt, you can press Esc to get from insert mode into command mode. There are some who will argue that in fact vi is modeless and that the “insert mode” is in fact a parameter to the insert command. This has a certain ring of validity to it – if you enter 32iHelloEsc in command mode, you will end up with 32 copies of the word “Hello” inserted.

But conventionally vi is written of as a mode-based editor, so it is best to think of it as such until you have learned enough to throw off conventional wisdom and go your own way.

The last mode is ex-mode, which at a basic level is covered only in enough detail to tell you how to get out of it! It is entered from command mode with the “:” command, at which point the cursor moves to the end of the screen leaving you free to type in lengthier commands. To exit simply hit the backspace key until the cursor returns to its normal location.

The Insert Mode

(and the replace mode)

The insert mode is started with a variety of commands, but the simplest is i(nsert).

Once in insert mode, you can start typing normal text without worrying about what commands it will run. There are a fair few things you can do with the control keys, but we’ll skip over those for the “basics”. To correct a few historical limitations :-

  1. You can move the cursor around with the arrow keys. It might seem a bit strange to say so, but the original vi didn’t allow you to move in insert mode partially because it pre-dates arrow keys (yes, really!) and had to use commands to move the cursor around the screen.
  2. You can move anywhere within the file and make changes anywhere; not just where the original change was intended. This may seem like an unnecessary feature to explain, but when you are changing a single word, it can seem wrong to also go somewhere else in the file and make changes elsewhere.

Without going into the more esoteric features, there is not a great deal more to say about the insert mode except it is exited with the Esc key.

The Command Mode

The movement commands :-

Arrow Keys Moves the cursor
h & l Moves the cursor left and right.
j & k Moves the cursor down and up.

The use of the h,j,k, and l keys to move the cursor around the screen seems rather bizarre except when you realise that some early terminals connected to Unix systems lacked cursor keys. They remain for compatibility reasons and because some people feel that they can be quicker to use as they require less hand movement than the cursor keys, or like me that those keys are burnt into muscle memory and so they are used almost without thought.

b(ack) Move backwards one word.
e(nd) Move forward to the end of the word.
f(orward){char} Move forwards on the same line to the next occurrence of {char}.
n(ext) Move to the next occurrence of the last search.
 / Search for something.

The most basic command for deleting text is “x” which deletes the character under the cursor, but a hint of what can be done with vi comes with the d(elete) command. The d(elete) command takes a movement as a parameter, and deletes from the current cursor position until where there movement takes you :-

dd Deletes line.
d$ Deletes to the end of the line.
d0 Deletes until the beginning of the line.
dw Delete until the end of the word.
diw Delete “in” word – deletes the current word.
df{char} Deletes until the next occurrence of {char}

But we now move finally to adding text :-

i(nsert) Insert at the current cursor.
o(pen) Open a new line below the current line.
O(pen) Open a new line above the current line.
a(ppend) Append text after the cursor.
A(ppend) Append new text at the end of the line.

Lastly, we can save and exit vi with “ZZ”.

Ex-Mode

This is going to be even more truncated than the last section (I know the last section doesn’t seem truncated, but trust me – it is!). There is a great deal more to this mode than just the three commands below :-

:write (or :w)

Writes the file being edited. Two options I am going to mention here.

Firstly you can add a filename to the command to write to an alternate file (:write new-filename) – very handy if you find you’re making changes to a file that you do not have permission to overwrite.

If you need to override a warning vi has about overwriting the current file, you can do so by appending an exclamation (!). Just don’t do it automatically (I’ve a sorry song to sing about doing that!).

:quit

And to quit vi, simply use the :quit command. If there are unsaved changes in the file you are editing, it will stop you, in which case if you really want to lose your changes add an exclamation (!).