Jun 212020
 

If you are just running Ubunbtu with ZFS without poking into the details, you may not be aware of the scrubber running. For background information, and for the benefit of those who prefer to go their own way, this is all about that little scrubber.

A pool scrub operation is where the kernel runs through checking all of the data in a pool and makes any necessary repairs. Whilst ZFS does check the integrity of the data (using checksums) when performing reads, a regular scrub repairs these issues in advance.

It need only be run weekly for larger systems or monthly for normal systems (it’s a pretty arbitrary border line). And can be started manually with :-

# zpool scrub pool0

(And “pool0” being the name of the pool to scrub)

Whilst a scrub is going on in the background, the only effect on the system is that disk accesses to that pool will be slightly slower than normal. Usually not enough to notice unless you are benchmarking!

When in progress the output of zpool status pool0 will show the current state and how long it is expected to take to complete the scrub. Once finished the status will look like :-

# zpool status | grep scan:
  scan: scrub repaired 0B in 0 days 09:19:27 with 0 errors on Sun Jun 21 10:36:28 2020

May 172020
 

There are two aspects to ZFS that I will be covering here – checksums and error-correcting memory. The first is a feature of ZFS itself; the second is a feature of the hardware that you are running and some claim that it is required for ZFS.

Checksums

By default ZFS keeps checksums of the blocks of data that it writes to later verify that the data block hasn’t been subject to silent corruption. If it detects corruption, it can use resilience (if any) to correct the corruption or it can indicate there’s a problem.

If you have only one disk and don’t ask to keep multiple copies of each block, then checksums will do little more than protect the most important metadata and tell you when things go wrong.

All that checksum calculation does make file operations slightly slower but frankly without benchmarks you are unlikely to notice. And it gives extra protection to your data.

For those who do not believe that silent data corruption exists, take a look at the relevant Wikipedia page. Everyone who has old enough files has come across occasional weird corruption in them, and whilst there are many possible causes, silent data corruption is certainly one of them.

Personally I feel like a probably unnoticeable loss of performance is more than balanced by greater data resilience.

Error-Correcting Memory

(Henceforth “ECC”)

I’m an enthusiast for ECC memory – my main workstation has a ton of it, and I’ve insisted on ECC memory for years. I’ve seen errors being corrected (although that was back when I was running an SGI Indigo2). Reliability is everything.

However there are those who will claim you cannot run ZFS without ECC memory. Or that ZFS without ECC is more dangerous than any other file system format without ECC.

Not really.

Part of the problem is that those with the most experience of ZFS are salty old Unix veterans who would are justifiably contemptuous of server hardware that lacks ECC memory (that includes me). We would no sooner consider running a serious file server on hardware that lacks ECC memory than rely on disk ‘reliability’ and not mirror or RAID those fallible pieces of spinning rust.

ZFS will run fine without ECC memory.

But will it make it worse?

It’s exceptionally unlikely – there are arguable examples of exceptionally esoteric failure conditions that may make things worse (the “scrub of death”) but I side with those who feel that such situations are not likely to occur in the real world.

And as always, why isn’t your data backed up anyway?

Apr 262020
 

Experimenting with Ubuntu’s “new” (relatively so) ZFS installation option is all very well, but encryption is not optional for a laptop that is taken around the place.

Whether I should have spent more time poking around the installer to find the option is a possibility, but post-install enabling encryption isn’t so difficult.

The first step is to create an encrypted filesystem – encryption only works on newly created filesystems and cannot be turned on later :-

zfs create -o encryption=on \
  -o keyformat=passphrase \
  rpool/USERDATA/ehome

You will be asked for the passphrase as it is created. Forgetting this is extremely inadvisable!

One created, reboot to check that :-

  1. You get prompted for the passphrase (as of Ubuntu 20.04 you do).
  2. That the encrypted filesystem gets mounted automatically (likewise).

At this point you should be able to create the filesystems for the relevant home directories :-

zfs create rpool/USERDATA/ehome/root
cd /root
rsync -arv . /ehome/root
cd /
zfs set mountpoint=/root rpool/USERDATA/ehome/root
(An error will result as there is something already there but it does the important bit)
zfs set mountpoint=none rpool/USERDATA/root_xyzzy
(A similar error)

Repeat this for each user on the system, and reboot. See if you can login and your files are present.

This leaves the old unencrypted home directories around (which can be removed with zfs destroy -r rpool/USERDATA/root_xyzzy). It is possible that this re-arrangement of how home directories work will break some of Ubuntu’s features – such as scheduled snapshots of home directories (which is why the destroy command needs the “-r” flag before).

But it’s getting there.

Apr 262020
 

A number of those who have experimented with Ubuntu’s ZFS install option (which as of 20.04 is marked as “experimental”) have expressed bewilderment over the number of filesystems created :-

The short answer as to why is that there are two schools of thought amongst grizzled old Unix veterans as to whether one big filesystem should be the way to go or lots of little ones. There are pros and cons to both approaches, and whilst I have a preference for lots of filesystems (especially on servers), I don’t care enough to change it on a laptop install.

Even though those who insist on one big filesystem are wrong.

As to the longer explanation …

Some History

A long time ago – the 1970s or the 1980s – Unix systems lacked sophisticated disk management software, and the disks were very much smaller (I started off with 80Mbyte disks and no that isn’t a typo, and many started with much smaller disks). On larger Unix servers, you couldn’t fit everything onto one disk, so we got used to splitting up the filesystem into many separate filesystem – / on one disk partition (or slice), /usr on another, /var on a third, /home on yet another, etc.

These very frequently got further subdivided – /var/mail, /var/tmp, /var/spool, etc. as Unix servers got larger and busier.

Those days are long past, and nobody is keen to go back to those days so why do some still like to split things up?

The Fringe Benefits of Splitting

It turns out that there was a fringe benefit to splitting up the filesystems – disk space exhaustion on one wouldn’t cause a problem elsewhere. For example if a mail server had a separate /var/spool/mail filesystem for operating within it would still continue to operate if /var filled up; similarly a DNS server wouldn’t crash and burn if it had a /var/named filesystem and /var filled up.

Both of those examples are known to me personally – and there are many other examples.

Of course there is also a downside – if you create a separate /var/spool/mail filesystem you need to make sure it is large enough to operate not just normally but in reasonable exceptional circumstances. Or your mail server crashes and burns.

On the other hand, if you don’t separate things out then when something goes berserk and fills up all the disk space then you will have a good deal of trouble actually logging in to fix things.

In a sense, the “everything in one” camp and “lots of little filesystems” camp are determined by what troubles we’ve seen over the years (and in some cases decades).

With something like ZFS you can set quotas to limit the size of any filesystem so managing the sizes of these separate filesystems is a great deal easier than it ever was in the past! Ubuntu does not set quotas by default on a desktop installation; for a server it may well be worth checking quotas and setting them appropriately.

And Snapshots …

One of the other things that Ubuntu does with ZFS and filesystem snapshots (we’ll worry about what those are another time) is to offer to rollback a broken update. People worry that upgrading their system will break things and the ability to quickly revert to the previous state is very comforting.

But the Unix file layout “standard” and the later Linux file layout standard were not designed with snapshots in mind, and simply rolling back the whole of “/” would have negative effects – not least you would lose any file changes you had made in /home and any mail stashed away in /var/mail.

So to implement the ability to rollback updates requires numerous separate filesystems to avoid losing important data.

It is also likely that it would be beneficial to tune separate filesystems for different requirements.

Finally

In short, don’t worry about it. It’ll have very little effect on your operation of a normal Ubuntu machine unless you choose to take advantage of it. And it makes possible certain features that you will probably like – such as the ability to revert updates.

Feb 082018
 

Some time ago, I wrote about using new (for the time) partition tables to create a memory stick with 100 partitions; each with a mountable file system on. And decided the time was right to have another look to see if things have improved … or degraded. After all, things have moved on, and everything has been updated.

I also improved the creation script slightly :-

#!/bin/zsh

disk=/dev/sdb

parted $disk mklabel gpt
for x in {1..99}     
do
  echo Partition: $x
  parted -s $disk mkpart FAT $(($x * 100)) $((x * 100 + 99))
  sleep 0.2
  mkfs -t vfat -n DOOM${x} ${disk}${x} 
  sleep 0.2
done

And I used a zsh-ism – so shoot me.

The script ran fairly well, but :-

  1. The load average shot up through the roof as copies of systemd-udevd started, worked, and closed.
  2. Strangely the links in /dev/disk/by-label (and presumably elsewhere) kept disappearing and re-appearing. As if on each partition change to the disk, all of the disk’s devices were removed and re-created. This is probably not dangerous, but harmful to performance.
  3. Given that I used sleep within my script, it is hard to criticise performance, but it did seem slow. However this is not an area worth optimising for.
  4. Unlike last time, Linux did not refuse to create any file systems.

Now onto trying to stick the memory stick of doom into various systems…

Ubuntu 17.10

This was of course the machine I ran the script on initially.

This did not go so well, with the machine initially freezing momentarily (although it is a cheap and nasty laptop), apparently silently refusing to mount half the file systems, and “Files” (or Nautilus) getting wedged at 100% processor usage.

After some 10 minutes, Nautilus was still stuck with no signs of making any progress.

After I lost patience and restarted “Files”, it came up okay showing the mounted file systems and showing the file systems it had failed to mount. On one occasion the additional file systems were shown as unmounted (and could be mounted) and on another they were shown as mounted (even though they weren’t).

So both “Files” gets a thumb down for getting stuck, and whatever else gets a thumb down for trying and failing (silently) to mount all the file systems.

This is definitely a serious degradation from the previous try, although probably GNOME-specific rather than Linux-specific. Especially as a later mounted all the file systems from the command-line on a different system without an issue.

Windows 10

Windows 10 became unusually sluggish, although it may have been in the mysterious “we’ll run Windows update at the most inconvenient time possible” mode. It did attempt to mount the file systems, and failed miserably – it mounted the first set until it ran out of drive letters.

Which is just about understandable, as there aren’t 100 drive letters. However :-

  1. Where was the message saying “There are 100 partitions in this silly USB stick. You can see the first 22; additional ones can be mounted within folders if there is important data on them.”.
  2. Why is Windows still limiting itself with single letter device names? Okay it is what we’re used to, but when you run out of drive letters, start using the file system label – “DOOM99:”. Hell, I’d like all my removable disks treated that way under Windows.

As for the whole “ran out of drive letters, so don’t bother with the rest”, how many people are aware that drives can be mounted (as Unix does) in directories?

macOS 10.13 (OSX)

Oddly enough (but perhaps sensibly), macOS refused to have anything to do with the memory stick. Indeed it popped up a dialog suggesting initialising the disk, which is perhaps not particularly sensible with a disk that could contain data.

The “Disk Utility” happily showed the disk – increasing the size of the window inconveniently wide in the process – and happily indicated 99 partitions.

At the Terminal prompt, it was apparent that the operating system had created device files for each of the partitions, but for some reason wouldn’t mount them.

Summary

Inserting a “stick of doom” with 100 partitions on it into any machine is still a risky thing to do. It’s also a dumb thing to do, but something operating system developers should be doing.

Linux (or rather GNOME) performs significant worse this time around than previously, and my suspicions are that systemd is to blame.

But however bad Linux does, none of the operating systems actually do sensible things with the “stick of doom”. macOS arguably comes closest with refusing to have anything to do with the disk, but it also encourages you to reformat the disk without saying that it could be erasing data.

Ideally, a gooey would pop up a window listing the file system labels and ask you which you want to mount. That’s not even a bad idea for a more sensibly set up memory stick.

Pebble On Steel

WP2Social Auto Publish Powered By : XYZScripts.com