Apr 062024
 

Just came across someone today who wasn’t aware of the “BCC” (Blind Carbon Copy) header, and was wondering how an email reached her when her address wasn’t in the “To” header. It’s all too easy to laugh at people who somehow missed learning this stuff, but how often does email get taught these days?

Headers Are Just Comments

Well that heading is a bit of an exaggeration but it’s a helpful exaggeration. It is perhaps more accurate to say the headers are hints to the underlying software. There is a chain of software “under the hood” that takes the email you have composed in some kind of email client (which includes a web mail interface which is the most common way these days), formats it into a suitable format for a “mail transport agent” which then determines the “mail transport agent” is closest to the recipients and sends it there.

You -> Mail client -> Your MTA -> Recipient’s MTA

In terms of headers that you populate to instruct that chain where emails should go, there is :-

  • The “To” header which is what is most commonly used.
  • The “Cc” (“carbon copy” – an archaic reference) header which allows you to specify additional recipients, but it implies that the additional recipients are included as a courtesy (“You might want to see a copy of this for information.”).
  • The “Bcc” (“Blind carbon copy” ) header, which allows you to specify additional recipients but when your client transfers your email to the mail transfer agent it will add the recipients to the “envelope” (which we will explain shortly) but remove the header.

There are two reasons for using “Bcc”. One is basic politeness – if you are sending to a lot of addresses, the recipients will see that header and it can take up valuable screen real estate distracting from the content of the email. The second is security – if you are sending an email to lots of third-party contacts it may well be appropriate (and even required) to hide their addresses from each other. Not everyone wants their relationship with an STD clinic to be “public”!

The “Envelope”

When a client communicates with the mail transport agent, it will use something called SMTP (simple mail transport agent) which is very simplistic and the MTA does not look at the contents to determine anything (or rather it does not need to; some do especially if they do anti-virus scanning) :-

Connected to peach.
Escape character is '^]'.
220 zonky.org ESMTP Exim 24.12 Sat, 06 Apr 2024 09:57:50 +0100
helo pica
250 zonky.org Hello pica.zonky.org [2001:8b0:ca2c:dead::b000]
mail from:<some-forged-address@zonky.org>
250 OK
rcpt to:<address1@zonky.org>
250 Accepted
rcpt to:<address2@zonky.org>
250 Accepted
data
354 Enter message, ending with "." on a line by itself
The email appears here including mail headers
.
250 OK id=1rt1ts-0001k8-MM
quit
221 zonky.org closing connection

That is a forged SMTP transaction with certain details changed. The important bits are in bold which are what your mail client would use to communicate with the mail transport agent. As you can see they are simple enough to be “faked” by a person. There is a great deal of trust going on here – far too much for the modern age – but there are additional controls in place to make forging things somewhat harder than this would imply.

The key commands are as follows :-

  1. mail from:<some-forged-address@zonky.org>: This specifies the address the email is apparently from. Normally this would be a setting in your mail client (whether you can change this or not), but there is nothing here to stop you setting any address you want. Although there are almost always additional controls in place to make this harder.
  2. rcpt to:<address1@zonky.org>. This specifies what email address the email should go to. It is usually pulled from the headers you filled in whether that was the To, CC, or BCC headers. At this stage there is no difference. However you can put in addresses that don’t appear in the email at all.
  3. data. This is where your mail client copies the email that has been composed including all the headers. It will remove the “BCC” header and add some additional ones (such as “Date”). This body may or may not be examined by the mail transport agent; it isn’t necessary to send the email onwards.

So the mail transport agent now has the necessary information it needs to route your email to the required destinations – without looking inside the body. Which is analogous to a letter – the Royal Mail doesn’t open your letter to see where it needs to go, they will just use the address on the envelope.

And so we have the explanation for an email envelope – it is the addresses specified in the SMTP transaction allowing the mail transport agent to route email without looking at the contents. In normal circumstances the mail transport agent for the recipient will discard the envelope before it is placed in the recipient’s mailbox.

Opening The Envelope

Just like real post where you have to trust that nobody along the route between the original writer and the recipient will open the envelope to peruse the contents, the same applies to email. Which all the ‘agents’ along the path can normally be trusted, there is nothing to stop a rogue agent examining the contents of email – whether that’s a snoopy system administrator, an employer with an overly suspicious nature, or law enforcement.

Which explains why it is strongly advisable not to use email for anything secret; or to investigate encrypting emails.

Rusty Handrail
Rusty Handrail
Jul 132019
 

This question popped on Twitter just now; I’m not sure the question was a serious one, but as it happens I can answer the question as if it were serious. I’ve been running relatively small mail servers for nearly 30 years, so do know a litle bit about it.

This of course does not cover lots of “bolt ons” to email such as SPF, DKIM, DMARC, and that’s just the simple stuff.

To keep things relatively sane, I’m going to gloss over an awful lot of details – for example, there are all sorts of ways of composing and sending emails, but I will assume you’re using some sort of web interface such as Google Mail.

So you compose an email with an interesting picture or two, and click “Send”.

Hang on a bit! You also did some technical stuff when you entered an email address into the “To” field (if you just enter someone’s name, something somewhere is changing that to an email address) and possibly something summarising the email (or trying to get attention) into the “Subject” field.

Headers

Those two headers you have filled in – ‘To’ and ‘Subject’ – are just two of a whole collection of headers that are usually invisible. They are invisible because your mail client is hiding them from you; for good reasons as they’re almost always an unnecessary distraction.

But when you have to diagnose an email problem, seeing the full set of raw headers is kind of essential even if it is dead simple (for the relatively technically sophisticated) to forge headers.

For example, if you set up a dedicated mail client rather than use your mail provider’s web-based mail client, you will have to fill out your own email address (to go into the “From” field). You can fill anything you like into that field although you are unlikely to get a reply

Email Addresses

If you fiddle with your mail client enough, you will realise that the “To” field can be changed to a “Cc” field or a “BCc” field. All three take email addresses, and can be used to send an email to someone but they operate in very slightly different ways.

The “To” header is for the main audience of a message.

The “Cc” header is for an audience who should also see the message, but perhaps don’t need to take any action based on the email.

The “Bcc” header is a bit special because it isn’t added to the raw mail message as a header, but is instead used to compose the envelope listing who should receive the email. If you need (or should) keep the list of people receiving an email secret, or just want to avoid an inconveniently long header for people to read then use the “Bcc” header.

Although your mail client may hide the details, email addresses look like :-

  1. some-name@example.com (a “pure” email address)
  2. “Some Name <some-name@example.com” (as normally formatted)

The Message Submission Server

So you click on “send”. What happens then?

The first thing that happens is that your mail client converts your email (attachments, rich formatting, etc.) into plain text which is why a file attachment just under your ISP’s limit on mail sizes can exceed that maximum.

To the top of that plain text version of your message, your mail client attaches some headers (again in plain text). In addition your mail client will prepare an “invisible” envelope independent of the headers – it contains just the address(es) you want to send to, and your own address.

Finally the mail client is ready to talk to the “mail submission server”. It will login (usually) with your credentials, and sends the message to the server using the “Simple Mail Transport Protocol” (SMTP).

The mail server adds your message to a queue and before it says to your mail client “Okay, I have it”, it will make sure that the message is safe on disk. Mail servers go to a lot of effort to make sure your messages don’t get “dropped on the floor” even when the operating system they are running on crashes.

The Journey

After you have submitted a mail message to the message submission server, the message is held in a queue. This is significant because email is a store and forward messaging solution, so every server your message journeys through will make sure it is stored on disk before saying to the server sending the message “Yep. Got it.”.

In theory this should mean that messages never get lost “in the system” (although practice is often different to theory). The store and forward style of architecture suits the era that email is from – back in the 1970s when email could (and did) traverse many different kinds of network rather than “the Internet” that we have today.

What does that mean today? Amongst other things, it means that whilst you may think that your message only visits two mail servers – the one belonging to your ISP and the one belonging to the recipient’s ISP – and that is certainly a valid architecture for very small organisations.

But it does not work so well for larger organisations that may handle millions, or hundreds of millions of message a day. Summing up a large architecture without actually seeing an architecture diagram for every single ISP and large organisation on the Internet is going to be a bit hit and miss, but I’ll give it a go.

Your message submission server will probably simply hand your message off to a server responsible for sending email messages across the Internet. If it cannot send to the recipient’s receiving mail server on the first run through the queue, it may well just hand the message off to a “slow message server” (processing large queues is a bit of a drag).

In either case, the recipient’s server will accept the message and then hand it over to a “mailbox server” that the recipient will connect to.

And the journey is over. At every stage, reliability is favoured over speed.

How do servers send your message to each other? Using that SMTP standard I mentioned earlier.

Speed

The speed of email is a good topic to cover here. Speed is not what email is about in the same way that the postal service for physical letters is not about speed. It’s about reliability

Your messages may be readable by the recipient within 5 minutes 99 times out of 100, but that 1 time may take a day or two. That’s just the way that email works – if you want instant messaging, use an instant messaging client.

Security

At no point have I mentioned encryption, and that is intentional. At no point is encryption required; email was designed in an era when computers were far slower and encryption was expensive (even though it was far weaker).

At every point, servers are trusting that the senders are not lying and the original sender is trusting that nobody is snooping on their emails.

To correct that there are a whole bunch of optional add-ons to the email standard to deal with that. But that is a story for another day.

End of the Pier