Aug 022014
 

One of the questions I always ask myself when setting up a resilient server, is just how well will it cope with a disk failure? Ultimately you cannot answer that without trying it out.

But as practice (and to determine whether it mostly works), it’s perfectly sensible to try it out on a virtual machine.

Debian Installation

If you are looking for full instructions on installing Debian, this is not the place to look. I configured the virtual machine with 2GBytes of memory, an LsiLogic SAS controller with two attached disks each of 64GBytes.

The installation process was much as per normal (I unselected “Desktop” to save time), but the storage was somewhat different :-

  • Manual partitioning method
  • Create an empty partition on both disks
  • Select Software RAID
  • Create an MD device
  • RAID1
  • And put both disks into the RAID
  • Configure LVM
  • Create a Volume Group (“sys”)
  • Select md0 for the volume group device
  • Create logical volumes (boot: 512MB, root: 16GB, var: 8GB, home: 512M (it’s a server))
  • In the partitioning manager select each Logical Volume in turn and specify the file system parameters.

You will notice that no swap was created – this was a mistake that I’m in the unfortunate habit of making! However for a test, it wasn’t a problem and with LVM it is possible to create swap after the installation.

Post Installation

After the server has booted, it is possible to check the second hard disk for the presence of grub in the MBR (dd if=/dev/sdb of=/var/tmp/sdb.boot bs=1M count=1, and then run strings on the result). It turns out that nothing is installed in the MBR of the second disk by default. Which would make booting in a degraded environment an interesting challenge (i.e. you’ll have to find a rescue CD and boot off the relevant hard disk).

However this can be fixed by installing grub onto the second hard disk: grub-install /dev/sdb

Testing Resilience

But what happens when you lose a disk? Now is the time to test. Shut down the virtual machine and remove the second hard disk – leaving the first hard disk in place does not provide a full test.

If your first attempt at booting afterwards results in a failure to acquire a grub menu, then either you have failed to run grub-install as detailed above (guess what mistake I made?), or your BIOS settings don’t permit the computer to boot off anything other than the first hard disk.

However, in my second attempt, the server booted normally with the addition of a few messages that indicate that there is just one disk making up the mirrored pair.

Summary

  1. Yes, you can put /boot onto an LVM file system that sits on mirrored disks. That hasn’t always been the case.
  2. It is still necessary to run grub-install to put Grub onto the MBR of the second hard disk.
  3. It works.
Jul 042014
 

Well, relatively safely anyway.

The other day I felt the urge to knock up some SQL to clean up a MySQL table I’m responsible for. So I got the SQL prepared, but then needed to find out how to run this as a cron job (or batch file if you prefer). This was because I wanted to run the job on a regular and frequent basis; and I’m too lazy to sit around with a stopwatch running the same command every 15 minute.

I came up with a method that I knew wasn’t recommended – as it said so in the manual. The method involved passing the contents of a file to the command line (i.e. mysql -u root -p$(cat somefile). And today I decided to have a look around to see if there was a better way.

It turns out that much of the most visible advice on the Internet is wrong. It all suggests using the -p option where the account password is visible in the process table (i.e. you can read it with ps aux). If you can guarantee that no rogue person will be able to login to your server and read the process table, then you are safe; and I would like to ask how you can achieve the impossible.

Putting a password in a file (as suggested below) is at least somewhat better, but it would be nice if MySQL were to offer something a bit more sophisticated.

However, on to the method.

The first step is to create an options file, and make sure that it is readable only by the current user :-

$ touch mysql-options-root.cnf
$ chmod go= mysql-options-root.cnf

Once securely created, you can edit the file and add the relevant options :-

[client]
password=do-you-really-think-i-would-tell-the-internet-what-my-mysql-root-password-is?

That list of options is fairly minimal; it only specifies the password to use. You can add additional options – just use the normal command-line options without the hyphens.

To use this options file, add the command-line option –defaults-extra-file=${filename}. A full example follows :-

$ mysql --defaults-extra-file=mysql-options-root.cnf --user=root --host=polio < use misc;
heredoc> select count(*) from films;
heredoc> END

The output is not included (it’s none of your business how many films I’ve watched). And yes this does work for what I intended it to.

As has been pointed out, it is also possible to schedule batch jobs within MySQL itself which may be a better answer in most cases.

Jul 032014
 

It is nice to make a shell environment more pleasant to use in many ways, but it is also helpful to ensure that the process degrades gracefully …

For example, I have a section in my .zshrc which creates an ls alias to use human-readable values, and to colourise the output :-

ls --color=auto > /dev/null 2>&1
#       Just collect the exit status ...
if [ "$?" = 0 ]
then
  # If there is no error then use the --color option
  alias ls='ls -h --color=auto'
else
  alias ls='ls -h'
fi

Thus when logging into a system that has an ls command that lacks the –color option, the alias will not create an ls command that immediately fails. Now whilst such systems are getting rather more rare than in the past, this graceful degradation is still useful as a principle. Whether creating shell aliases, or more generally.

As another example, I have a shell alias (page or also pg) that I use to invoke a “pager” like more, less, pg, or my preferred choice, most. The relevant section within the zshrc file is :-

for candidate in more less most
do
  p=$(which $candidate)
  if [ "$?" = "0" ]
  then
    alias pg=$candidate
    alias page=$candidate
    PAGER=$candidate
  fi
done

This repeatedly sets up the two aliases (and sets the PAGER environment variable) if the candidate pager is available; otherwise the aliases are left alone. In other words, this works through a list of candidates in order from most available to least available (but preferred) to select one. Once used to using page as a command, I no longer need to worry about if most is installed on a system.

A very similar loop is used to generate an alias called vim which will always work (at least when vi is available).
 

Jun 262014
 

Came across a hint today about reporting on ECC memory errors. For those who do not know, ECC memory detects memory errors and corrects correctable errors. Normal memory (as found in almost all laptops and desktops) simply ignores the errors and lets them accumulate and cause problems either with data corruption or by causing software errors.

As I happen to have ECC memory in my desktop machine I thought I would have a look into the hint. Turns out that Linux does not report on ECC events automatically; you need to install the relevant EDAC (Error Detection and Correction) tools. Which for Debian, turns out to be pretty simple :-

# apt-get install edac-utils

As part of the installation process, a daemon process is started. But for whatever reason, it didn’t automatically detect what driver to load. So I edited /etc/default/edac and added :-

EDAC_DRIVER=amd64_edac_mod

Once that is done, a simple /etc/init.d/edac restart loads the driver and starts monitoring. Messages should appear in your log files (/var/log/messages) and reports can be displayed with edac-util :-

# edac-util --report=full 
mc0:csrow0:mc#0csrow#0channel#0:CE:0
mc0:csrow0:mc#0csrow#0channel#1:CE:0
mc0:csrow1:mc#0csrow#1channel#0:CE:0
mc0:csrow1:mc#0csrow#1channel#1:CE:0
mc0:csrow2:mc#0csrow#2channel#0:CE:0
mc0:csrow2:mc#0csrow#2channel#1:CE:0
mc0:csrow3:mc#0csrow#3channel#0:CE:0
mc0:csrow3:mc#0csrow#3channel#1:CE:0
mc0:noinfo:all:UE:0
mc0:noinfo:all:CE:0

Of course memory errors are relatively rare (or at least should be) so it may take months before any error is reported.

Jun 192014
 

… or a red ✗. Incidentally, if your browser doesn’t show ticks (✓), crosses (✗), and a right pointing double arrow (») properly, this posting may look a bit odd.

As in :-

2014-06-19_1934

The aim here is to make the zsh prompt (a Unix shell) start with a green ✓ (tick) if the previous command’s exit status was zero, and a red ✗ (cross) if the previous command’s exit status was non-zero. A tiny thing, but both useful and fun.

The first thing I need is a set of variables containing terminal sequences for setting colours. Fortunately zsh comes with a set of suitable functions; even if they’re poorly named :-

autoload -U colors && colors

The next thing is to perform a test to see if the current terminal type is worth setting a fancy prompt for. In some cases – such as when using the plain Linux console, or when things are broken enough that the terminal type isn’t set properly – it is worth avoiding setting a prompt. I do this by setting up an array containing a list of terminal types that I think it is worth setting a fancy prompt for :-

fancyterms=()
fancyterms=(xterm xterms dtterm iris-ansi xterm-256color)

(Some of that list is very historical!)

The test for whether to set a fancy prompt is somewhat tricky … it uses a ‘reverse subscripting’ flag in the array lookup to search the array for the string :-

if [[ ${fancyterms[(r)$TERM]} == "$TERM" ]] 
then
  echo Fancy prompt
else
  echo Plain prompt
fi

And lastly, I actually set the prompt :-

export PROMPT="%(?.%{$fg[green]%}✓ %{$reset_color%}.%{$fg[red]%}✗ %{$reset_color%})%B%n@%m»%b "

Which is not exactly the easiest string to understand, but breaking it up :-

  1. The %{$fg[green]%} sequence sets the colour to green, the ✓ is fairly self-explanatory, and the %{$reset_color%} sets the colours back to normal.
  2. Similarly the %{$fg[red]%} sets the colour to red, the ✗ should also be self-explanatory, and the %{reset_color%} does as before.
  3. The sequence %{ … %} is very important as any output that does not advance the cursor should be contained within these. This allows zsh to count the number of visible characters within the prompt so that various screen operations happen in the right location. Judging by how often this is emphasised (and the fact that I made a mistake with it myself), it looks to be a very common problem.
  4. The sequence %{?.True.False%} tests the exit status of the previous command and if true outputs the first string and if false outputs the second string.
  5. The sequence at the end – %B%n@%m»%b – turns bold on (%B), outputs the username (%n), outputs an “@”, outputs the short machine name (%m), outputs a literal “»”, turns off cold (%b), and finally adds a space to the prompt.

Putting it all together, I get :-

# Now the prompt
autoload -U colors && colors
#       Enable colour variables (used in prompt)
fancyterms=()
fancyterms=(xterm xterms dtterm iris-ansi xterm-256color)
#       An array of 'fancy' terminal types that we do more for … as in a funky PROMPT.

if [[ ${fancyterms[(r)$TERM]} == "$TERM" ]] 
then
        precmd () { print -Pn '\e]2;%n@%m - %~^G' }
        # Put the hostname in the Window title/Tab title
        export PROMPT="%(?.%{$fg[green]%}✓ %{$reset_color%}.%{$fg[red]%}✗ %{$reset_color%})%B%n@%m»%b "
        # And set a really fancy prompt.
else
        # If we don't recognise the terminal type, don't attempt to be quite so ambitious
        # with the prompt.
        export PROMPT=$'%B%n@%m - %~\n%#%b '
fi

This is still a little bit plain compared with what some people do with their prompts, but it is right for me (at least for now).