LVM – More Zonkyness

Ubuntu Server: The Interactive Installer Sucks Big

Jun 092024

So on Friday my workstation blew up … which goes some way to explaining why this server has been down for much of the weekend (it’s a container on my workstation). The operating system boot drive magically went read-only – which as some of us know is a clear indication that an SSD is on its last legs. Or a few steps beyond.

So I re-installed on a new drive, and for various reasons I chose Ubuntu Server again. An upgrade which made things a bit more interesting.

Now whilst I know that most servers are installed in ways other than interactively, but the interactive experience is bloody awful.

For a start, if the text console is a ridiculous large size – perhaps $COLUMNS is greater than 160, then change the bloody console font. It may be a rare case that someone is installing Ubuntu server on a screen where the resolution is so high, but it can (and in my case does) make the text far too small to read.

And secondly, do something about the logical volume manager creation; I wanted to create a logical volume with a name other than “ubuntu-vg” (the old drive was still readable and creating two VGs with the same name struck me as a dumb idea). The default method didn’t seem to offer a way to rename the VG; the custom method kept giving me an installation error.

The later could possibly be my mistake – I was reduced to using a magnifying glass.

And yes everything is working now.

But sort out that installer!

Debian: Testing The Resilience Of A Mirrored Installation

Computing, Linux, Working Notes No Responses »

Aug 022014

One of the questions I always ask myself when setting up a resilient server, is just how well will it cope with a disk failure? Ultimately you cannot answer that without trying it out.

But as practice (and to determine whether it mostly works), it’s perfectly sensible to try it out on a virtual machine.

Debian Installation

If you are looking for full instructions on installing Debian, this is not the place to look. I configured the virtual machine with 2GBytes of memory, an LsiLogic SAS controller with two attached disks each of 64GBytes.

The installation process was much as per normal (I unselected “Desktop” to save time), but the storage was somewhat different :-

Manual partitioning method
Create an empty partition on both disks
Select Software RAID
Create an MD device
RAID1
And put both disks into the RAID
Configure LVM
Create a Volume Group (“sys”)
Select md0 for the volume group device
Create logical volumes (boot: 512MB, root: 16GB, var: 8GB, home: 512M (it’s a server))
In the partitioning manager select each Logical Volume in turn and specify the file system parameters.

You will notice that no swap was created – this was a mistake that I’m in the unfortunate habit of making! However for a test, it wasn’t a problem and with LVM it is possible to create swap after the installation.

Post Installation

After the server has booted, it is possible to check the second hard disk for the presence of grub in the MBR (dd if=/dev/sdb of=/var/tmp/sdb.boot bs=1M count=1, and then run strings on the result). It turns out that nothing is installed in the MBR of the second disk by default. Which would make booting in a degraded environment an interesting challenge (i.e. you’ll have to find a rescue CD and boot off the relevant hard disk).

However this can be fixed by installing grub onto the second hard disk: grub-install /dev/sdb

Testing Resilience

But what happens when you lose a disk? Now is the time to test. Shut down the virtual machine and remove the second hard disk – leaving the first hard disk in place does not provide a full test.

If your first attempt at booting afterwards results in a failure to acquire a grub menu, then either you have failed to run grub-install as detailed above (guess what mistake I made?), or your BIOS settings don’t permit the computer to boot off anything other than the first hard disk.

However, in my second attempt, the server booted normally with the addition of a few messages that indicate that there is just one disk making up the mirrored pair.

Summary

Yes, you can put /boot onto an LVM file system that sits on mirrored disks. That hasn’t always been the case.
It is still necessary to run grub-install to put Grub onto the MBR of the second hard disk.
It works.

Linux Logical Volumes Not “Active” ?

Linux, Working Notes No Responses »

Aug 202013

Every so often I come across an old Linux box that doesn’t take kindly to being rebooted. Without console access, it is hard to see what is going on, but the Linux kernel gets stuck trying to mount the root file system. There are many possible fixes for this, but they all have one thing in common … a work-around has to be performed to get the box up and running.

The console gets stuck in a “mini-root” environment loaded when the initrd image is loaded and before the real root file system is mounted which means a lot of commands are not available, but lvm should be available. First of all, run lvm lvscan to get a list of the logical volumes that need activating :-

(initramfs) lvm lvscan
  inactive          '/dev/sys/root' [332.00 MiB] inherit
  inactive          '/dev/sys/usr' [8.38 GiB] inherit
  inactive          '/dev/sys/var' [2.79 GiB] inherit
  inactive          '/dev/sys/swap_1' [7.05 GiB] inherit
  inactive          '/dev/sys/tmp' [380.00 MiB] inherit
  inactive          '/dev/sys/home' [16.00 GiB] inherit
  inactive          '/dev/sys/opt' [24.00 GiB] inherit

For each volume group (the second column, middle word), run: lvm lvchange -ay ${volume-group-name}. In the case of my example :-

(initramfs) lvm vgchange -ay /dev/sys
  7 logical volume(s) in volume group "sys" now active

At which point you should be able to press ^D (or enter exit) to continue the boot process.

A slighter better work-around involves changing the Grub configuration to add a delay to the kernel parameters. This sections assumes that you are not using Grub Legacy!

Start by editing /etc/default/grub and changing the variable GRUB_CMDLINE_LINUX to include “rootdelay=20” :-

GRUB_CMDLINE_LINUX='console=tty0 console=ttyS0,19200n8 rootdelay=20'

Finalise by running update-grub. This adds a 20s delay to the boot process so is hardly an ideal solution.

Linux Volume Manager: Growing Filesystems

Computing, Linux, Working Notes No Responses »

Feb 062013

If you have previously used Linux’s volume manager (LVM) to set up disk storage, you may want to know about how to grow a filesystem safely.

Which is probably the big feature of any decent volume manager because accurately predicting the size of filesystems is a black art, and the only alternative – to make the root filesystem contain all of the storage is a dumb idea.

It’s actually really easy and can be done non-disruptively. It is done in two parts – effectively growing the “disk” device and then growing the filesystem itself.

Extending The Volume

First identify the volume you need to extend. You can of course simply run lvscan which will show a list of the volumes, and if you have named them sensibly will allow you to pick out the volume to extend. But the simplest way is simply to run df to look at the filesystem you want to extend :-

/dev/mapper/ssd-opt         7.9G  5.5G  2.1G  73% /opt

The device (in the first column) is what we extend. Now to decide how much to grow the volume by; just for the case of this example, we’ll assume that 2Gbytes is a sensible amount to grow the volume by. The command needed is :-

lvextend --size +2G /dev/mapper/ssd-opt

And that’s it. No need to shut down the server, dismount the filesystem, etc. Of course we haven’t quite finished yet.

Growing The Filesystem

What we have done at this point is the equivalent of making the disk bigger. We also need to tell the filesystem it is sitting on a bigger disk, and to do so we need to know the type of the filesystem. The canonical place for checking that is the file /etc/fstab (actually it’s the filesystem itself but that is going too far) :-

# grep opt /etc/fstab
/dev/mapper/ssd-opt	/opt		ext4	noatime		0 3

It is probable that you are looking at growing an ext3, ext4, or xfs filesystem. If not you will have to look up the details yourself.

Growing ext3, or ext4 Filesystems

This is done with the resize2fs command :-

resize2fs /dev/mapper/ssd-opt

Several points :-

Yes it can be done “online” whilst the filesystem is mounted (and applications are busy using it).
You need to specify the device containing the filesystem to grow and not the mount point.
There is no need to specify the size … the size will be determined from the size of the device underneath the filesystem.

Growing xfs Filesystems

This is done with the xfs_growfs command :-

# xfs_growfs /opt

Several points :-

Yes, it can be done “online”. In fact you have to do it with the filesystem mounted.
You need to specify the mountpoint of the filesystem and not the device. Irritatingly different from the above!
There is no need to specify the size.

How Reliable Is This?

Very.

There is always the chance that something could go wrong especially if you are operating “at the edge” (say you have a filesystem that is unusually large – several petabytes). But I’ve done online filesystem resizing for years in countless circumstances without an issue.

I’ll quite happily do it on the most important systems during working hours without losing a moment’s thought. However I do work in a place that takes backups seriously!

The Move To ArchLinux

Computing No Responses »

Nov 202010

For some time now, I have been contemplating switching Linux distributions on my main workstation from Ubuntu to something a little less … user friendly ? Or perhaps that should be a little more Unix geek friendly. The distribution I chose was ArchLinux for a variety of reasons. If you come across this blog entry looking for a solution to a problem, it may be worth reading through in case the solution appears later on – this is long, and searches may “hit” on something later on.

First of all, let me point out there is really nothing wrong with Ubuntu for most users. It is a pretty useful distribution that is pretty good for the kind of users who have never compiled their own kernel. Nothing wrong with that, but it seems that Ubuntu is gradually becoming a little trickier to use for those of us who prefer to customise their desktop environment with something like Enlightenment – it seems that Ubuntu is really intended for those who want the Ubuntu way.

Nothing wrong with that, and I’m intending to keep running Ubuntu on my netbook. However I wanted a little more control for my main workstation. And what with an SSD to install as a new boot device, it seemed like a good time to try out ArchLinux especially as I could reboot into Ubuntu if things looked bad. As it happens I haven’t needed to do that! This blog entry is going to get quite long as a place to record my notes on getting ArchLinux to do the things I want, and will grow over time.

The Install

I downloaded the core install image rather than the net install image – not for any good reason as I have done test installs from the net install image and it works well. After installing the SSD into my workstation (stuck to the bottom of the case with duct tape – I should really get a 2.5->3.5″ disk tray), I changed the boot order of the disks in my BIOS to boot from the SSD first. This was perhaps not the best idea as it made things a little trickier later, but it’s workable if you are prepared to juggle disk names (both Linux ones and BIOS/Grub ones).

First for the boring bit :-

Booted off the install CD
Selected CD as source
Set Europe/London as timezone
Set hwclock as UTC
Prep hard drives-
1. Manually configure hard drives
2. Partition /dev/sdc (the SSD – identified by the fact it was empty)
3. Created 256Mb partition /dev/sdc1 (for /boot)
4. Created partition with the rest of the space /dev/sdc2 as LVM
5. Manually configure block devices
  1. By device name
  2. Created /boot on /dev/sdc1 as ext2
  3. /dev/sdc2 becomes Volume Group
  4. / as XFS (16G)
  5. /var as ResierFSS (4G)
  6. swap (4G) – Although I have a tendency to forget this one!
  7. /opt as XFS (4G)
  8. /tmp as ReiserFS (4G) – perhaps a bit too big.
Select Packages
1. Select Base + Development.
2. Pick random additions that look like they might be useful (note that it may be necessary to pick all of the various mkinitcpio variations as I did that on the later attempts).
Install Packages
Configure System
1. Select ‘vi’ as editor
2. Made the following changes to rc.conf
  1. UseLVM=yes
  2. HOSTNAME=scrofula
  3. eth0=”eth0 10.0.0.18 netmask 255.255.0.0 broadcast 10.0.255.255′
  4. gateway=”default gw 10.0.0.254″
  5. ROUTES=(gateway)
3. Made the following changes to mkinitcpio.conf
  1. BINARIES=”/sbin/lvm”. This shouldn’t be necessary, but at one point I ended up with a miniroot shell which was unable to mount the root filesystem and with no LVM present, I couldn’t see what was wrong! This error could be related to the raid problems detailed below, but adding this won’t cause any harm.
  2. HOOKS=”base udev autodetect scsi sata lvm2 filesystems”. Note that “raid” is suggested as necessary for software RAID; that turns out to be incorrect as discovered later. Although I needed software RAID to mount my /home, I left that for later after putting raid in here gave errors)
4. Made the following changes to resolv.conf
  1. search inside.zonky.org
  2. nameserver 10.0.0.12
5. Made the following changes to mirrorlist
  1. Select something from “Great Britain”.
6. Set root password.
7. Done
Install Bootloader
1. Grub
2. Installed to /dev/sdc! This is because although the SSD is the third by address, it is also the first boot device in the BIOS.

This didn’t work the first time around. Firstly grub wasn’t setup properly as it wanted to boot the next stage from (hd2,0) which would be one of the hard disks rather than the SSD, as at this point the BIOS is still in charge (more or less). This was easily fixed on a temporary basis by editing the boot setting at the menu, and later on a more permanent basis by editing /boot/grub/menu.lst.

Secondly the first couple of times around, I found myself in what I term the “miniroot shell” which is the shell you get when the Linux install fails to mount the root filesystem. The only hint I had here was that a) it couldn’t mount the root filesystem, and b) the binary /bin/lvm was not present. On the third or fourth attempt (my notes aren’t sufficiently accurate) I managed to get past this stage by excluding the raid “hook” and including the /bin/lvm binary in the mkinitcpio configuration file.

It would seem that at some point ArchLinux has changed the “hook” name from raid to dmraid and some instructions out there still refer to the hook as “raid”. My fault for not checking closely enough with enough sources! But there’s no harm in the ArchLinux people configuring both names – probably just a case of setting up a hard link somewhere!

Post-Installation

With a distribution such as ArchLinux, the easy part is the installation; things get a bit trickier with the post-installation configuration. This is simply because to allow you to do things your way, it needs to leave things alone and let you do your stuff. In other words this lack of default configuration is a feature and not a bug!

The first thing to so after a core install (and probably a net install too) is to perform a full update :-

pacman -Suy

The “pacman” tool is of course the ArchLinux package management tool. This operation sits somewhere between a normal Ubuntu package upgrade and a full Ubuntu distribution upgrade. ArchLinux does not have distribution versions in the same way as Ubuntu – whilst the installation media is undoubtedly upgraded from time to time, once actually installed the command above will perform both upgrades to apply necessary fixes, and upgrade packages when new versions come out.

This can lead to some surprises from time to time of course, but there is also never quite the same level of shock that comes with a distribution upgrade.

In any case, I needed to run the command twice as pacman itself needed an upgrade.

After doing that, I set CONSOLEFONT in /etc/rc.conf to “sun12x22.psfu” to improve the appearance of the console, although there are another couple of fonts based on that font that may well be a better choice. Later I used the “consolefont” hook to set the console font at an earlier stage during the boot process – which is neater; however you should specify the font without the file extension – “sun12x22”, and of course add “consolefont” to the HOOKS variable in /etc/mkinitcpio.conf.

I also edited /boot/grub/menu.lst to change the line that specifies what kernel to load and it’s options :-

kernel /vmlinuz26 root=/dev/mapper/ssd-root ro vga=775

Specifically adding the “vga=775″ to the end of that. This makes the appearance of the console not quite so overwhelming on a 30” monitor!

Also added “dmraid” to the HOOKS variable in /etc/mkinitcpio.conf although reading more documentation hints that the right hook is actually “mdadm”. Run mkinitcpio -p kernel26 to update things.

Rebooted to verify that things are still working. Plus check that the CONSOLEFONT was ok, and that the old volume group:sys was visible.

Older Entries