Feb 272017

Strictly speaking how some cloud services do mail wrong, but whilst it is not all, there are still quite a few that do which is why there are no names contained within this rant.

When you have some cloud-based service send email, it makes sense for the “From” header (i.e. what sensible normal people think of as the sender address) to contain the email address of the person using the cloud-based service.

Fair enough.

But if the real sender address or envelope sender address (which is contained within the SMTP transaction) comprises the email address of the person using the cloud-based service you may well run into problems. Many organisations publish an SPF record in their DNS to indicate what network addresses are approved for, and many mail servers check the envelope sender against the published authorised network addresses.

If the network address used by the cloud service provider does not match what is in the organisation’s SPF record then the recipient’s mail server is free to reject the mail. And they often do.

Now the most obvious “fix” for this is to add the cloud service provider’s network address to the organisation’s SPF record.

The only trouble with that is that it isn’t always possible. There are various limits to how long an SPF record can be so adding addresses to the SPF record recklessly is unwise, and a sensible DNS administrator will only add to the SPF record for important services. So if the cloud service is being evaluated or being used by something less important, or is being used for non-work related purposes, then it likely won’t meet the “important enough to get added to the SPF record” criteria.

So why not fix the source of the problem?

All that has to be done is to use a different address for the envelope sender, and you can even arrange things to send bounces back to the right place.

Set the envelope sender to something like “customer+${original email}@${cloud service address}” (obviously when replacing the ${original email} you will have to change the “@” sign to something reversible). All of a sudden you are no longer “forging” the envelope sender, and not tripping over anyone’s spam defences.

Process the bounces to “customer@${cloud service address}” and you can send the bounces to the right place.

Apr 022014

Not that many images but the idea is to go for quality and not quantity. Boring old black and white too. And no, there’s no colour versions of these.

#1: The Three Scarecrows of Morecambe

The Three Scarecrows of Morecambe

#2: Across The Bay

Across The Bay

#3: The Bird

The Bird

Yes there is a bird flying within this image.

#4: Disappearing Sea

The Disappearing Sea

#5: The Bare Family

The Bare Family

To “get” the pun in the title, it is worth pointing out that this statue was found in Bare … a sub-district of Morecambe.

#6: The Stone Thumb

The Stone Thumb

Oct 152012

Backups are funny things … everyone says that they’re important, but actually it is restores that are important. Backups merely make restores possible. Because restores are so infrequent (and frankly backups so boring) there are far too many of us who do not spend enough time making sure everything is backed up properly.

This is not a blog entry saying how backups should be performed in the technical sense … I’m not going to suggest you use rsync to an offsite cloud storage provider, or how frequently the backups should be made. But rather a random wandering around the problems of backups.

Before I forget, I will be using a shortcut: When I say “to tape”, it merely means copying the backup to whatever medium you use. I’ve merely been doing backups long enough that “to tape” just flows naturally. In fact my current backup media is an external hard disk attached to an off-site server.

What Do You Want To Restore?

Before you can decide what you need to backup, you need to decide what you might want to restore. And what types of data you might want to restore. You can split up restores into three broad categories – the system data, the user data, and any databases there might be. Each has slightly different requirements – you might well back them all up in the same way, but it may also be better to back them up in three different ways.

Restoring Systems Data

If you backup the operating system of your computer using the same mechanism intended for backing up user data, then in a disaster situation you will be faced with the interesting situation that the data you need to make your computer functional again will be sat on a tape, an external disk, or (even worse) sitting on a patch of this cloud stuff that is currently so trendy.

Whilst this is perhaps not the ideal situation to be in, it is also not the end of the world. At least if you are not pushed for time to get the computer back up and running again.

There are basically two options here – either a dedicated computer imaging solution that clones your operating system disk in some way, or to use the original installation DVDs as a restoration method. The later may be the cheap option, but it does work to the extent of at least getting you going again. And of course lets you into those other backups you have made.

The decision on which to go for comes down to time – how long would it be acceptable for you to be without your data? Bearing in mind that the restore can only start after you have the hardware to restore it to.

Restoring Databases

When it comes to restoring from backups, databases can be a touch fragile. However it is worth pointing out that by this I mean real databases such as Oracle, PostgreSQL, or MySQL rather than database applications such as Microsoft Access. If you just copy the files making up a database to a backup tape, then the result can probably be restored but you may well end up with a corrupt database, and it may be missing some data.

The classic method of backing up a database is to shut it down, and then copy the files to tape. That is a pretty safe way of doing things, but if you are trying to run a 24×7 service, then it is rather inconvenient. That is not to say it is not still a good method – simply accept the need to shutdown the database once a week and concentrate on methods to minimise that downtime (filesystem snapshots work brilliantly here).

There are also database specific methods of generating a backup whilst the database is running. The lowest common denominator here is the equivalent of an export – the database generates a whole bunch of SQL commands which when run re-create the database. These methods can be used in combination with the old shutdown and copy to tape mechanisms to double your chances of getting a good backup.

And indeed allow you to minimise the disruption by only performing a shut down and copy to tape less frequently than every night.

Of course you probably do not have a database at home that you need to keep running 24×7, but some people will. But even if you don’t care how often the database gets taken down to perform a backup, you still should spend some time making sure that your database is backed up properly. It is too late to check when you are trying to perform a restore.

Oh! And if you do shut down your database to back it up, please remember to start it again afterwards!

Restoring Ordinary Files

Backing up ordinary files is definitely on the most boring side of backups. But for most of us, they are the most important backups we perform; as more and more of our important data becomes digital rather than physical, we need to be sure that our digital data is safe. And safe does not mean just safe from the odd hard disk failure, but from disasters such as house fires!

Or from foolish decisions to delete the wrong files, etc. We tend to assume that restores are performed after disasters such as the aforementioned hard disk crash, but in practice – at least in an organisation with a team responsible for performing restores – files are restored almost continuously and far more frequently than hard disk crashes.

You can choose to backup everything – which means you can be sure that you have everything you need to restore in an emergency, although it can be a lot slower as you will be backing up a lot of “junk”, or you can be very selective in what you backup which makes things a lot quicker, but there is always the danger that something important will be lost.

Or you could do both! It is perfectly sensible to backup only the most important files every day – perhaps to DropBox – and then do a full backup once a week.

One thing to look for is something along the lines of Apple’s Time Machine; there are approximate equivalents for Windows and Linux, and the key advantage that all of them has, is the ability to perform differential backups which means that only the changed files are copied. My own backup script ran last night and ‘refreshed’ a backup of nearly 500Gbytes in about 7 minutes (and that was to a very remote server).

And use those backups! Checking whether the backups have worked or not is another tedious job, but using yesterday´s files is far less tedious.

A Few Misconceptions

  1. RAID isn’t a backup method. You can mirror your hard disks (I do), but that merely reduces the probability of a hard disk crash causing you to reach for the backup tapes. That is not to say that it is worthless, but you still need to perform backups.
  2. Tape isn’t dead. It may well be too expensive for home use, but tape is still a perfectly sensible way of keeping backups. It can be “enhanced” in various ways such as snapshots to give the impression of backups being performed very quickly, or a disk buffer to keep the most recent backup online.
  3. Cloud backup solutions are cool, but not without issues. For a start you have to worry about the legal aspects (especially if you are a business), such as whether the backups are stored within an acceptable jurisdiction. In addition, what happens if the cloud storage provider goes out of business for some reason ? There are quite a few people who could tell you the problems of using certain cloud storage providers who have for one reason or another ceased operation.
  4. A backup on the same disk as the source files may well be a poorly considered option. After all, it will not help you if the hard disk goes “bang”. But it could be quite a good supplementary option to another method. Similarly an external disk is all very well, but will not help you much in the case of a house fire.
  5. Whatever backup method you choose is subject to failure. The external hard disk that fails just when you need it, the encrypted cloud backup where you’ve forgotten the passphrase because it’s held in a password store on the disk you’re trying to recover, etc. Having multiple backup destinations is worth considering especially when so many cloud storage providers are giving so much space away for free.
May 022012

One of the cool things about “the cloud” is that there are numerous different companies all offering cloud-based storage of one kind or another. You can even get quite a bit of storage for free, and different solutions offer different cool solutions – such as Dropbox where my phone is configured to automatically send photos up to it. And there are plenty of other solutions out there :-

  • Box
  • Google Drive (of course you may already be using Google Docs which means you essentially have storage related to that).
  • SkyDrive (although for some mysterious reason, Microsoft doesn’t supply a Linux client)
  • iCloud
  • Wuala
  • SpiderOak
  • Ubuntu One – which despite the name, isn’t just for Ubuntu!
  • And in a note for myself, there’s also SparkleShare which is essentially a DropBox client to talk to your own servers.
Undoubtedly there are a whole ton more, but I think I’ve gotten the “big names” covered. The best strategy is of course to find the one whose client works with all the platforms you use (phone, PC, laptop, etc.), comes with the most free storage, and the cost of getting more storage is the least (in decreasing order of importance). Of course in the real world, you are likely to end up with more than one – simply because it’s tempting to look at the next “new thing” or because you want more cheap storage, or simply because other people insist you use service X.

Now if you use multiple cloud-storage solutions, you have a bit of a problem – different clients offering different functionality, different amounts of storage available, and remembering what you put on which “cloud-disk”. Plus of course there is the interesting problem of security – different providers provide different levels of privacy and operate in different jurisdictions where different laws apply.

Different Clients

Different clients work in different ways with different features. For instance, for a Linux user :-

  1. The Dropbox client seems to work pretty well, but it doesn’t appear in a list of filesystems (i.e. when you type df) so you can’t instantly see how much space is still available, etc. At least not in the standard way.
  2. Box(.net) lacks a Linux client, so you have to hack something together. Perfectly possible for more geeky users, but even for us there is the danger that a hackish solution may suddenly stop working mysteriously. Or rather that is more likely.
  3. Ubuntu One doesn’t seem to work via a filesystem interface at all.
  4. And that seems to be the same with SpiderOak.
It may be different for Windows users (I’m too lazy to check – if anyone wants to submit details, please go ahead), but I doubt it.

Whilst cloud storage providers may offer additional features to differentiate their product, they are all essentially the same as a removable hard disk, usb memory stick, or some other kind of removable storage. Whilst the additional features are very welcome, why should we have to learn a new way of managing storage just because it is out there in the cloud ?


There is a great deal of paranoia about storing private data in the cloud with the assumption that creepy organisations such as Google will do something nasty with the data. Well maybe, but the likelihood of Google being that interested in an individual’s data is a little unlikely. Of  course just because the cryptogeeks are a little paranoid does not mean they are completely wrong – there are privacy issues involved.

Firstly, Google could be looking at your data to determine things about you that would be of interest to advertisers – to present targeted adverts at you. Which at best can be a little weird.

Next we like to believe that the laws of our country will protect us from someone picking through our personal data. That someone could be the company supplying the storage, or it could be the government in the country where the storage is hosted. That would probably be fine if the storage was restricted to one location where we could be sure that the government protected us, but where is the storage located?

Much of the time the storage is located in foreign jurisdictions where there is no guarantee that any kind of privacy will be respected – especially if a foreign government takes an interest in your data. Don’t forget the laws of say the USA are not designed to protect citizens of any EU country (or visa-versa). There are of course agreements such as the EU Safe Harbour agreement, but it is possible that it does not offer as much protection as assumed – it is not really intended for private individuals choosing to put their own personal data into foreign jurisdictions.

Probably most of us do not have to worry about this sort of thing (although we can choose to), but some may have to be cautious about this sort of thing. Some of us deal with personal data about third parties – sometimes very personal data – and need to consider whether storing such data in the cloud is being appropriately responsible about the data privacy. For example, a contractor who stores information about their clients should be taking actions to ensure that data is not accidentally leaked (or hacked and published).

The easy answer to this problem is to assume that cloud storage is not safe for sensitive personal data, because there is a simple solution to the problem that still allows the cloud to be used. Use encryption such as TrueCrypt to ensure that even if the cloud leaks your data, it is still encrypted with a method that is not known to the cloud provider.

Store It Twice!

There have been occasions where storage providers have removed access to storage either permanently or temporarily – such as the Megauploads site. Whilst it is perhaps unlikely, it is possible for a cloud service provider to disappear and for the customers to lose their data – even if the cloud provider claims that there is some protection against this sort of thing happening. But it could happen, so it is sensible to ensure that if you store data in the cloud, that you should ensure that you have copies of that data elsewhere.


WP2FB Auto Publish Powered By : XYZScripts.com

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.