Feb 082009
 

I was reading a comment about the df command (in relation to reserved filesystem space) and realised that the clueless newbie was right; it is odd that df does not mention reserved space. Of course it would also be wrong for df to lie about the matter too. I then realised that df is long overdue for a bit of refreshing. If you look at the typical output of the df command, you will find it inconveniently cluttered :-

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/datavg-810root
                       12G  7.8G  4.3G  65% /
tmpfs                 2.0G     0  2.0G   0% /lib/init/rw
varrun                2.0G  416K  2.0G   1% /var/run
varlock               2.0G     0  2.0G   0% /var/lock
udev                  2.0G  3.1M  2.0G   1% /dev
tmpfs                 2.0G  344K  2.0G   1% /dev/shm
lrm                   2.0G  2.4M  2.0G   1% /lib/modules/2.6.27-7-generic/volatile
/dev/sdb1             130M   36M   88M  29% /boot
/dev/mapper/datavg-opt
                      2.0G  776M  1.3G  39% /opt
/dev/mapper/datavg-810var
                      5.0G  1.4G  3.7G  28% /var
/dev/mapper/datavg-home
                      256G  116G  141G  46% /home
/dev/mapper/datavg-vmachines
                       96G   62G   35G  64% /vmachines
/dev/mapper/datavg-bragspool
                      256G  6.2G  250G   3% /var/spool/brag
/dev/mapper/datavg-herpesbackup
                       16G  4.6G   12G  29% /var/herpes
/dev/sda1             463G  147G  293G  34% /mdata
/dev/scd0             2.4G  2.4G     0 100% /media/CIVCOMPLETEEU
/dev/mapper/datavg-cdimages
                       32G  1.9G   31G   6% /cdimages
/dev/mapper/datavg-ontapsim
                       16G  498M   16G   4% /sim

Part of the problem is that df does not do quite what it claims to do … to report free space on the mounted filesystems. It also gives some (a very small amount) of additional information about the relevant filesystems … particularly the device the filesystem is mounted on. This “helps” to make the output more cluttered that it needs to be. It is possible that there are those who will argue that the device is the filesystem and not where it is mounted; they are arguably right, but when you use df you are either looking at where in the Unix file hierarchy there are places that have less space than is comfortable, or for places that have enough space to put that big file you are about to download.

Next the command itself has an obscure command to make it easier to type on a slow type-writter like terminal (those who are below a certain age will not realise that we used to comminicate with Unix machines using a terminal that was more like a printer than the screens we use today). It might be better named fsspace with an alias of diskspace for those who want to concentrate on what worries them rather than on what worries the machine.

Next why not take advantage of certain features that have crept almost silently into the command line over the last few decades ? Why not adjust the output to the width of the terminal window (look for the $COLUMNS evironment variable), spacing things out or even adding more information when you have enough space?

Finally if you were to dig around the df command a little bit you will encounter something peculiar called “inodes”. Now I know what an inode is, and I dare say quite a few people reading this will know, but if you do not, knowing how many inodes there are is not very useful information. It is relatively rare (these days) for a filesystem to run out of inodes so this information has a low priority, and why not use a term more understandable than “inodes” ?

Changing a term is something to be avoided in most circumstances which is why we still have “inode” where even the originator of the term has to guess that the “i” means “index”. I would suggest that something like “fileslots”or perhaps “fslots”

We now have the basic specification of something that should look like :-

% diskspace
Filesystem                            Size  %Used %fslots  Avail
/                                      12G    70%      3%   3.6G
/lib/init/rw                          2.0G     0%      0%   2.0G
/var/run                              2.0G     0%      0%   2.0G
/var/lock                             2.0G     0%      0%   2.0G
/dev                                  2.0G     0%      1%   2.0G
/dev/shm                              2.0G     0%      0%   2.0G
/lib/modules/2.6.27-7-generic/vola+   2.0G     0%      0%   2.0G
/boot                                 130M    29%      0%    88M
/opt                                  2.0G    40%      1%   1.2G
/var                                  5.0G    18%      0%   4.1G
/home                                 256G    52%      0%   124G
/cdimages                              32G    65%      0%    12G
/mdata                                463G    36%      1%   280G

This could be improved in some ways – for instance it would be helpful to skip over certain of the filesystems that are not strictly speaking backed by disk. However it is beginning to be useful.

Or would be if the code exists. Fortunately it does.

Dec 052008
 

I recently encountered a dead blog entitled “Linux Haters” and instantly thought up writing about tedious fan-boys that think that the operating system they like is the best and everyone should use it. I’ve no time for people like that as they tend to annoy rather than educate. I’ve no problem with people who prefer to use Windows, Linux, Solaris or OSX; it is their choice. Of course in the case of Windows, I do have to wonder why 🙂

But one of the links on that blog led to a place that (amongst other things) ranted about how FOSS projects always have dumb names, and that these projects need a big dose of marketing intelligence. He went on to whinge about the word-games often embedded into the project name.

First of all, he misunderstands how many open source projects start – with a geek or a group of geeks deciding they want something different. Either a new package or a variation on an existing one. There are no marketing types in sight, and the geeks involved probably have no great expectation that they are coming up with the next big thing – they are just having fun and hoping to come up with something useful for themselves. So what if they have a bit of fun playing word games to come up with a name for their project ? Not only do many such projects end up disappearing without a trace, but as marketing types have fun playing with words, why can’t geeks ?

Perhaps the names they come up with are not as punchy as a name thought up by a marketing department, but weirdness does have its own value in this area. A name such as Amarok does tend to stick in the mind more than Music Player 52. And over time, formally weird names such as google and yahoo do tend to become more normal if they are attached to popular projects.

Secondly he specifically criticises names invented by geeks for being recursive acronyms … but does that matter ? He specifically names GIMP which is admittedly particularly guilty being a recursive acronym with no termination. But most users won’t care … once they learn that GIMP does images (and most distributions will tell you so in the menu), they are not going to care that the name is an infinitely recursive acronym … they will just get on and use it.

Thirdly he overlooks the fact that some of the names may in fact have “sensible” names but are in fact sensible names in non-English languages.

Finally he tails off into a moderately incoherant rant with more insults than proper facts.

Perhaps “funny” names do put people off, but perhaps not. Most people are in fact more concerned with compatibility (they use Word because everyone else does) or features.

And of course there are more than a few commercial software packages whose name is not entirely sensible … does Photoshop have anything to do with setting up a shop to sell photos? What does Trent do ? Or Cedar ?

Nov 082008
 

Well I decided it was time to upgrade my main workstation to Ubuntu 8.10 from 8.04. This was a somewhat nervous upgrade because the SATA disks are connected to a controller that gives me trouble when booting … the last few times I rebooted the machine I have had to re-assemble one of the mirrors at the “miniroot” (the shell you get from the initrd environment).

Given the trouble I wanted the option to revert back to the current install if everything went wrong … something I have described in the past.

Environment

I have four SATA disks in my main workstation that are mirrored and encapsulated into a single volume group with numerous logical volumes. So /dev/md0 is constructed of a mirror of /dev/sda2 and /dev/sdb2 and /dev/md1 is constructed of a mirror of /dev/sdc1 and /dev/sdd1. /dev/sda1 is used as /boot and /dev/sdb1 is “spare”.

The volume group datavg has two “physical disks” in it – /dev/md0 and /dev/md1. In that volume group are (amongst others) two logical volumes of interest – 804root and 804var.

Preparation

First I ensured that /dev/sdb1 contained an up to date copy of /boot :-

mkfs -t ext2 /dev/sdb1
mount /dev/sdb1 /mnt
star -copy /boot /mnt
umount /mnt

The next step was to create two new logical volumes to be used (eventually) for the new release of Ubuntu :-

lvcreate --size 12G -name 810root /dev/datavg
lvcreate --size 3G --name 810var /dev/datavg
mkfs -t xfs /dev/datavg/810root
mkfs -t xfs /dev/datavg/810var

The next step was to copy the filesystems across – at this point no further non-upgrade activity took place :-

mount /dev/datavg/810var /mnt
star -copy -xdev /var /mnt
mount /dev/datavg/810root /mnt
star -copy -xdev / /mnt
(edited /mnt/etc/fstab to change references to 804 to 810)
umount /mnt

At this point the remaining task (or so I thought) was to change the references in /boot/grub/menu.lst so that Grub would boot the kernel with 810root as the root filesystem.

I then rebooted to check the functionality and discovered that for some reason /var/run/network had not been created on the new /var but that seemed to be the only issue. I re-created this directory and rebooted to be sure (and Linux without a loopback device behaves very strangely!).

I then in theory had an environment I could revert to with a bit of fiddling with grub so was happy to attempt the upgrade.

The Upgrade Itself

I then started the upgrade (sudo update-manager -d) only to be told I did not have enough disk space with the suggestion that I run sudo apt-get clean to free up enough space. I tried this and the update started on the next time through. Why doesn’t the upgrade process automatically run apt-get clean ? In fact why doesn’t the upgrade process realise that it is on a system with LVM available (with plenty of free space) and create a new LVM especially for the upgrade files ?

Anyway the second attempt to run update-manager did start downloading files so I settled down for a well earned beer.

After the files had downloaded the actual upgrade process began. At some point the process stopped whilst a certain upgtade (libpam) asked whether it should restart certain services (including gdm which is the graphical login). Unfortunately it did not explain clearly what it was doing, why, or even if restarting gdm would terminate the current X session. Perhaps it is something that would only “worry” someone with a dangerous amount of knowledge, but perhaps the prompt could be improved slightly.

As a later example of how it should be done, the dialog warning about changing the ssl certificate was ideal … basically saying “if this doesn’t mean much to you, don’t worry about it”.

The use of dialogs during the upgrade does raise a couple of issues though :-

  1. Apparently not “stopping the clock” whilst waiting for the user to respond to dialogs caused the estimate of how long the upgrade would take place to increase dramatically. Just after dealing with that libpam prompt I was slightly alarmed by the estimate of 6 further hours to complete (as it was already 21:30) only to be reassured by the rapid drop to 2 hours.
  2. From a users perspective it would be preferable to ask all questions at the beginning or at the end of a long process. Keeping an eye on the upgrade whilst it was churning through the bulk of the upgrade (which took place without user intervention) was somewhat tedious.

The unfortunate thing was that at this point, the upgrade process ran out of space. It also left my machine in an unbootable state … from what little investigation I made it would appear that the initrd file was not created. Perhaps because of the lack of space, but others have also encountered this problem.

Obviously the Ubuntu upgrade process should be a little more careful about estimating the amount of space available. It would also be nice if it would notice that it was installed on an LVMed system with free space available … it could create a new LVM of an appropriate size and use that.

Recovery

Fortunately my preparation left me with a root filesystem that I could revert to by selecting one of the older kernels in the grub menu, editing the command used to boot, and by replacing the root filesystem with “804root”. This resulted in a system almost identical to that before the upgrade process.

I re-tried the “Preparation” stage, ensuring that /var was a little bigger and restarted the upgrade process …

Upgrading (The Second Attempt)

The second upgrade went through pretty much the same process as the first … except that it did not run out of disk space. This let it complete apparently normally with a warning about removing several hundred packages (!).

Finally it suggested rebooting to finish the upgrade.

The First Boot

The first boot was a triffle problematic … firstly I still had to manually assemble /dev/md0 and the process that built /boot/grub/menu.lst hadn’t noticed that an invalid root filesystem was specified. Whilst the later was my fault, it would be nice if it had realised that it was invalid and had warned me. The former is a long standing problem and is not the result of the upgrade.

Apart from that, the standard user interface looked ok, but it had messed around with the Enlightenment window manager. I had to re-install the package, my settings and menus had disappeared. Whilst I am an “unusual” user in not using the standard user interface, this was not something I was happy with!

Aug 252007
 

If you’re hoping to read about Linux finally getting ZFS (except as a FUSE module) then you are going to be disappointed … this is merely a rant about the foolishness shown by the open-source world. It seems that the reason we won’t see ZFS in the Linux kernel is not because of technical issues but because of licensing issues … the two open-source licenses (GPL and CDDL) are allegedly incompatible!

Now some may wonder why ZFS is so great given that most of the features are available in other storage/filesystem solutions. Well as an old Unix systems administrator, I have seen many different storage and filesystem solutions over time … Veritas, Solaris Volume Manager, the AIX logical volume manager, Linux software RAID, Linux LVM, …, and none come as close to perfection as ZFS. In particular ZFS is insanely simple to manage, and those who have never managed a server with hundreds of disks may not appreciate just how desireable this simplicity is.

Lets take a relatively common example from Linux; we have two disks and no RAID controller so it makes sense to use Linux software RAID to create a virtual disk that is a mirror of the two physical disks. Not a difficult task. Now we want to split that disk up into seperate virtual disks to put filesystems on; we don’t know how large the different filesystems will become so we need to have some facility to grow and shrink those virtual disks. So we use LVM and make that software RAID virtual disk into an LVM “physical volume”, add the “physical volume” to a volume group, and finally create “logical volumes” for each filesystem we want. Then of course we need to put a filesystem on each “logical volume”. None of these steps are particularly difficult, but there are 5 seperate steps, and the separate software components are isolated from each other … which imposes some limitations.

Now imagine doing the same thing with ZFS … we create a storage pool consisting of two mirrored physical disks with a single command. This storage pool is automatically mounted as a filesystem ready for immediate use. If we need separate filesystems, we can create each with a single command. Now we come to the advantages … filesystem ‘snapshots’ are almost instantaneous and do not consume additional disk space until changes are made to the original filesystem at which point the increase in size is directly proportional to the changes made. Each ZFS filesystem shares the storage pool with the size being totally dynamic (by default) so that you do not have a set size reserved for each filesystem … essentially the free space on every single filesystem is available to all filesystems.

So what is the reason for not having ZFS under Linux ? It is open-source so it is technically possible to add to the Linux kernel. It has already been added to the FreeBSD kernel (in “-CURRENT”) and will shortly be added to the released version of OSX. Allegedly because the license is incompatible. The ZFS code from Sun is licensed under the CDDL license and the Linux kernel is licensed under the GPL license. I’m not sure how they are incompatible because frankly I have better things to do with my time than read license small-print and try to determine the effects.

But Linux (reluctantly admittedly) allows binary kernel modules to be loaded into the kernel and the license on those certainly isn’t the GPL! So why is not possible to allow GPLed code and CDDLed code to co-exist peacefully ? After all it seems that if ZFS were compiled as a kernel module and released as a binary blob, it could then be used … which is insane!

The suspicion I have is that there is a certain amount of “not invented here” going on.

Apr 282007
 

This is intended to be quite a long piece and may be saved before it is fully completed. Some of the content will be more general ramblings on Linux in general rather than specific to Ubuntu 7.04

I recently installed Ubuntu 7.04 (not without a few problems that most people are unlikely to encounter) and thought it worth rambling through a few thoughts that occurred to me. Most of the ideas are related to how Ubuntu would come across to less experienced users although to be frank I find it difficult to put myself in their shoes. One point to make fairly early is that Linux distributions have conflicting goals … they need to appeal to the less experienced user without putting off the ones who have been running Linux for years, compiling their own kernels and generally getting used to the deeper levels of Linux. One idea here is to have two “sides” to every configuration screen … one for the easy options and one for the advanced options. This does not necessarily need to be implemented as a GUI window that can be turned over, although that is not a bad idea.

I installed using the “alternate” installation CD and did not bother with the “user-friendly” partitioning options, so I can’t say much about the normal installation CD or the partitioning experience. However it is worth noting that partitioning is a somewhat tricky concept to someone new to Linux who has not necessarily done much in the way of partitioning under Windows. Also selecting different filesystem types (ext2, reiserfs, xfs, jfs, etc.) is not something that the average user will be comfortable with.

Mind you Linux installation is not exactly difficult. Those who claim it is, are frequently overlooking just how difficult XP is to install. Either they are already used to it, or have never installed it in the past! Of course installing Linux is trickier than running the Windows that was installed in the factory on the average PC.

Incidentally, whilst I understand that setting up complex partitions and filesystems is inevitably going to take a while, my configuration took as long to setup as the rest of the installation! This is somewhat extreme!

So the ordinary user has managed to work their way through the installation routine and has rebooted the system. They are now faced with a blank screen with just a ‘username’ prompt in the middle. It would be nice here to have a one-off prompt in a seperate window here to explain that the user needs to login with the username and password they created during the installation and a brief explanation of why logging in is a wise mechanism

So the user logs in and is presented with a fairly typical GNOME screen which is quite blank. It would be quite nice to start a “What To Do Now” screen here. The GNOME help feature is quite useful when started, but it should be started for new users. However advanced users may prefer to “turn it over” to get a more complex default index with content that applies more to them. This could be nothing more than a single item on the help screen titled “Advanced Users” to give instructions on how to do it (something like dpk-reconfigure yelp advanced-view) … a command line command here is appropriate to indicate a barrier that should be climbed before it is appropriate to “turn this page over”.

One other thing on a series of documents explaining Linux to new users. It could explain some of the typically installed applications … which ones to use to do what, and where to find them in the menus. It could also explain the reason for the funny names … that many of the applications are created by programmers and named by them, and not to let their sometimes peculiar sense of humour be offputting.

Traditionally Ubuntu has avoided including proprietry codecs for common multimedia formats such as MP3. This is for genuinely legitimate reasons and I am not suggesting this changes, but the help screens should have a number of entries relating to this. “Playing Proprietry Encoded Music And Videos” should point to a help screen saying why these formats are not included by default and simple instructions in installing them. “Why Can’t I Play MP3s” and “Why Can’t I Play DVDs” should give a short explanation and point to the instructions on installing them. By all means make the point that priorietry formats are evil, but be helpful too.

Oh! And don’t lie in help screens. At one point Ubuntu claims that their package manager is the only way to install software. This is obviously not true to advanced users and could eventually be seen as not true to ordinary users too. Just say that it is strongly recommended to install software using the package manager as an obvious mistruth makes one wonder what else is wrong.

Ubuntu comes with a fairly easy way of enabling “desktop effect” with an appropriate warning about their stability. However it only enables compwiz and I wanted to have a quick look at beryl. This was acomplished fairly easily, and I suddenly had access to a great deal more desktop effects. Some very interesting eyecandy it was too, some of which I can see could be quite useful.

However the preferences screen was a little swamped with different configuration options. Whilst beryl is most definitely in an early phase of development, it would be wise to look at this. Not that all of the options should be removed, but going back to a phrase I used earlier, “turning over” the options screen to keep the advanced options hidden from most users should be considered. It also needs far more explanation of what all the options are. Perhaps a button to “grow” a simple explanation into a longer more detailed explanation.

What if things go wrong and the new user needs assistence ? Well there are two parts to this … problems during the installation that results in a system that cannot be booted, and problems that crop up after installation.

For the second, there needs to be a section in the help screens on obtaining assistence. This should assistence in obtaining information about the broken system (perhaps Linux needs a tool like “Sun Explorer” which generates a compressed archive containing the output of many different diagnostic commands such as fdisk -l, cat /proc/cpuinfo, etc). Also explaining how best to phrase support requests … anyone who has done technical support knows the problems that can come about because of badly expressed problem issues.

Ubuntu helpfully has pointers to sources for free online support, and to commercially available support. However it would be useful pointing out the basic difference between the two … free support can be of as high a quality as commercial support, but you cannot be certain of getting a response. Whereas commercial support has the downside that it costs money.

Finally (well … if you are lucky 🙂 ), Ubuntu comes with a fine graphical package manager called Synaptic; whilst as a crusty old Unix veteran I prefer the command-line equivalent, it does do a pretty good job. However a new user looking at Synaptic could be a little overwhelmed by the number of packages that are available. Synaptic has a series of ways of viewing the package repositories which can be helpful in finding what you want; why not add an additional default view (with a prominent button saying “See the rest”) that has just one (well perhaps up to five) “best of breed” package listed for each application.

A new user is less likely to be overwhelmed when installing software if they visit the package manager and see “3D Modeler” -> “Blender” instead of the current situation where “Blender” itself consists of half a dozen packages that they need to hunt through several hundred applications to find.