Jul 032014
 

It is nice to make a shell environment more pleasant to use in many ways, but it is also helpful to ensure that the process degrades gracefully …

For example, I have a section in my .zshrc which creates an ls alias to use human-readable values, and to colourise the output :-

ls --color=auto > /dev/null 2>&1
#       Just collect the exit status ...
if [ "$?" = 0 ]
then
  # If there is no error then use the --color option
  alias ls='ls -h --color=auto'
else
  alias ls='ls -h'
fi

Thus when logging into a system that has an ls command that lacks the –color option, the alias will not create an ls command that immediately fails. Now whilst such systems are getting rather more rare than in the past, this graceful degradation is still useful as a principle. Whether creating shell aliases, or more generally.

As another example, I have a shell alias (page or also pg) that I use to invoke a “pager” like more, less, pg, or my preferred choice, most. The relevant section within the zshrc file is :-

for candidate in more less most
do
  p=$(which $candidate)
  if [ "$?" = "0" ]
  then
    alias pg=$candidate
    alias page=$candidate
    PAGER=$candidate
  fi
done

This repeatedly sets up the two aliases (and sets the PAGER environment variable) if the candidate pager is available; otherwise the aliases are left alone. In other words, this works through a list of candidates in order from most available to least available (but preferred) to select one. Once used to using page as a command, I no longer need to worry about if most is installed on a system.

A very similar loop is used to generate an alias called vim which will always work (at least when vi is available).
 

Jun 262014
 

Came across a hint today about reporting on ECC memory errors. For those who do not know, ECC memory detects memory errors and corrects correctable errors. Normal memory (as found in almost all laptops and desktops) simply ignores the errors and lets them accumulate and cause problems either with data corruption or by causing software errors.

As I happen to have ECC memory in my desktop machine I thought I would have a look into the hint. Turns out that Linux does not report on ECC events automatically; you need to install the relevant EDAC (Error Detection and Correction) tools. Which for Debian, turns out to be pretty simple :-

# apt-get install edac-utils

As part of the installation process, a daemon process is started. But for whatever reason, it didn’t automatically detect what driver to load. So I edited /etc/default/edac and added :-

EDAC_DRIVER=amd64_edac_mod

Once that is done, a simple /etc/init.d/edac restart loads the driver and starts monitoring. Messages should appear in your log files (/var/log/messages) and reports can be displayed with edac-util :-

# edac-util --report=full 
mc0:csrow0:mc#0csrow#0channel#0:CE:0
mc0:csrow0:mc#0csrow#0channel#1:CE:0
mc0:csrow1:mc#0csrow#1channel#0:CE:0
mc0:csrow1:mc#0csrow#1channel#1:CE:0
mc0:csrow2:mc#0csrow#2channel#0:CE:0
mc0:csrow2:mc#0csrow#2channel#1:CE:0
mc0:csrow3:mc#0csrow#3channel#0:CE:0
mc0:csrow3:mc#0csrow#3channel#1:CE:0
mc0:noinfo:all:UE:0
mc0:noinfo:all:CE:0

Of course memory errors are relatively rare (or at least should be) so it may take months before any error is reported.

Jun 192014
 

… or a red ✗. Incidentally, if your browser doesn’t show ticks (✓), crosses (✗), and a right pointing double arrow (») properly, this posting may look a bit odd.

As in :-

2014-06-19_1934

The aim here is to make the zsh prompt (a Unix shell) start with a green ✓ (tick) if the previous command’s exit status was zero, and a red ✗ (cross) if the previous command’s exit status was non-zero. A tiny thing, but both useful and fun.

The first thing I need is a set of variables containing terminal sequences for setting colours. Fortunately zsh comes with a set of suitable functions; even if they’re poorly named :-

autoload -U colors && colors

The next thing is to perform a test to see if the current terminal type is worth setting a fancy prompt for. In some cases – such as when using the plain Linux console, or when things are broken enough that the terminal type isn’t set properly – it is worth avoiding setting a prompt. I do this by setting up an array containing a list of terminal types that I think it is worth setting a fancy prompt for :-

fancyterms=()
fancyterms=(xterm xterms dtterm iris-ansi xterm-256color)

(Some of that list is very historical!)

The test for whether to set a fancy prompt is somewhat tricky … it uses a ‘reverse subscripting’ flag in the array lookup to search the array for the string :-

if [[ ${fancyterms[(r)$TERM]} == "$TERM" ]] 
then
  echo Fancy prompt
else
  echo Plain prompt
fi

And lastly, I actually set the prompt :-

export PROMPT="%(?.%{$fg[green]%}✓ %{$reset_color%}.%{$fg[red]%}✗ %{$reset_color%})%B%n@%m»%b "

Which is not exactly the easiest string to understand, but breaking it up :-

  1. The %{$fg[green]%} sequence sets the colour to green, the ✓ is fairly self-explanatory, and the %{$reset_color%} sets the colours back to normal.
  2. Similarly the %{$fg[red]%} sets the colour to red, the ✗ should also be self-explanatory, and the %{reset_color%} does as before.
  3. The sequence %{ … %} is very important as any output that does not advance the cursor should be contained within these. This allows zsh to count the number of visible characters within the prompt so that various screen operations happen in the right location. Judging by how often this is emphasised (and the fact that I made a mistake with it myself), it looks to be a very common problem.
  4. The sequence %{?.True.False%} tests the exit status of the previous command and if true outputs the first string and if false outputs the second string.
  5. The sequence at the end – %B%n@%m»%b – turns bold on (%B), outputs the username (%n), outputs an “@”, outputs the short machine name (%m), outputs a literal “»”, turns off cold (%b), and finally adds a space to the prompt.

Putting it all together, I get :-

# Now the prompt
autoload -U colors && colors
#       Enable colour variables (used in prompt)
fancyterms=()
fancyterms=(xterm xterms dtterm iris-ansi xterm-256color)
#       An array of 'fancy' terminal types that we do more for … as in a funky PROMPT.

if [[ ${fancyterms[(r)$TERM]} == "$TERM" ]] 
then
        precmd () { print -Pn '\e]2;%n@%m - %~^G' }
        # Put the hostname in the Window title/Tab title
        export PROMPT="%(?.%{$fg[green]%}✓ %{$reset_color%}.%{$fg[red]%}✗ %{$reset_color%})%B%n@%m»%b "
        # And set a really fancy prompt.
else
        # If we don't recognise the terminal type, don't attempt to be quite so ambitious
        # with the prompt.
        export PROMPT=$'%B%n@%m - %~\n%#%b '
fi

This is still a little bit plain compared with what some people do with their prompts, but it is right for me (at least for now).
 

Jun 052014
 

There are many different ways to listen to music, and many different ways to listen. The old way to record music is on huge discs containing an analogue recording of the music on both sides – what is now called the Long-Playing record (or LP). It’s called long-playing because it is an evolution of other disc-based formats that were capable of holding less music.

The not quite so old method of holding music is the Compact-Disc (or CD) which uses stores the music digitally rather than in analogue form.

The newest method is the “digital download” and is itself as complex as all the previous methods put together. Because the playback of digital music recordings is basically software, it is possible to introduce new “formats” every so often – so we have MP3, Ogg, FLAC, etc. Most of these formats compromise on music quality in favour of smaller file size so they can be downloaded quicker and you can fit more of them on the average portable music player.

However for the purposes of comparing with LPs and CDs, I will only consider high-resolution FLAC files which do not compromise on quality over file size. Specifically FLAC files with a better than CD quality, and yes they do exist.

When considering music playback, it’s worth remembering that good music always sounds better than bad music; an improved playback merely makes good music sound better. And there are diminishing returns.

Having said that, which of the three is better: LPs, CDs, or digital downloads? The answer is complex, and depends upon the what you want to get out of an album. It also depends on the manufacturing quality – a poorly pressed CD on inferior materials does not compare well with a good CD; and perhaps even more so for an LP.

The Physical Experience and Longevity

In terms of an object to be appreciated in addition to the music nothing beats the LP. Excluding the disc itself, the album art is large enough to be fully appreciated. This is for many an important consideration, but personally I buy recorded music for the music and not for the object d’art that it comes on.

In terms of longevity, nothing lasts forever, but a DRM-free digital download is perhaps somewhat surprisingly in the lead here. Providing that you have a good backup regime, there is no reason why a digital download cannot be passed on to your off-spring.

On the other hand, LPs are subject to various forms of damage that can occur by chance every time you play one – even the playback causes a very tiny amount of damage unless you use a laser-based record player. And every bit of damage (or dust) causes a deterioration in the quality of the playback.

CDs have certain advantages in this respect because it is digital and there is built-in error correction so smaller errors get corrected.

However neither physical medium will last forever, and a poorly treated LP is likely to deteriorate significantly in a relatively short amount of time.

Analog vs Digital

very strongly suspect that the preference that some people have for analog has nothing to do with the fact that it is analog. If you play back a quality LP through a high-end system, it may seem to sound slightly better than a CD. But I have listened to very high quality digital rips of LPs played back on a high quality digital music player, and have heard the same effect – even including the added “warmth” that LP playback supposedly gives you.

analog-digital

The graph above is a very exaggerated demonstration of the difference between a “perfect” analog signal (which frankly doesn’t exist) and a very rough and ready digital signal. The distance between the green lines is representative of the sample rate – how many times per second a digital value is measured. The vertical scale of the green lines is representative of the number of bits used to measure each sample – in this case it is far too few leading to a dramatic staircase effect.

The digital version can be improved by increasing the sample rate and increasing the number of bits for each sample. Increase it enough, and it is impossible to distinguish between the digital and analog versions.

But increase the sample rate to 192KHz and the resolution to 32-bits, and you are talking about some really rather large files. Which is why CDs are just 44KHz and 16-bits; at the time CDs were developed, it was felt to be a reasonable compromise between quality and the need to keep storage requirements down.

One thing that is often overlooked is that CDs contain a lot of error correcting data – 20% of the storage capacity is error correction data. This allows the CD player to correct any read errors that occur (up to a certain level). Nothing else has this level of error correction with the exception of a digital download if it is held on an error-correcting file system (such as ZFS).

This section implies that analog is “perfect”. In case anyone believes that, try reading up on wow, and flutter. And indeed the full details of analog vs. digital. Whilst analog in general is no worse than digital, the specific analog method used for LPs is inherently less detailed than the amount of detail available in CD quality audio and indeed high-resolution digital downloads.

The interesting thing about the so-called “warmth” that LPs provide is that this is effectively making a virtue of the weakness of LPs. The old terms for LP warmth are “muddy bass” and “rolled off high end”. Listening to the warmth of LPs may well be a better experience but that does not make LPs better for music playback.

Digital Compression

Judging by some of the snake oil around digital music, compression is not very well understood – for example. there is a fairly popular myth that the audio from FLAC can be improved by uncompressing into WAV format and then playing it back. There is possibly a very tiny advantage in doing that if you happen to have very poor quality digital equipment, but the advantage would be miniscule. That advantage would be simply down to the hardware/software being unable to uncompress and play back the audio stream at the correct speed all of the time.

There are essentially two different forms of digital compression – lossless and lossy. The former can be uncompressed to produce a digital stream exactly the same as the original, and the latter does not. Lossy compression is much smaller in size, so it is better suited to small portable music players, although there is no reason why small portable music players have to have such small capacities for storage (my current portable player has 256Gbytes of storage and my previous portable player had 240Gbytes of storage).

Because lossy compression loses some of the original data, it approximates the original data – ideally in a way that is indistinguishable from the original. However it is possible that extreme compression – as found in 128kbps MP3 files – will introduce artifacts that sound “wrong”.

Summary

When you come down to it, music is not about digital or analog, it is about the music. And to many quality is also a desirable aspect. This can be obtained with analog or digital, but to a great extent digital is more convenient.

If you are after quality, there is no need to switch to analog to get it – digital music can be just as filled with quality as analog providing that you do it properly.

But most importantly, enjoy the music!

May 192014
 

So there I was, contemplating whether I could produce a nice and simple bar chart showing the number of films I’ve watched per month. Using just MySQL.

I knew it was easy to produce a simple numerical count with count(*) and a group by clause, but a bar chart? Turns out it was easy with the repeat string function.

mysql> select date_format(whence, "%Y-%m") "Month", 
          repeat('*', count(*)) "Number of Films" 
          from films group by date_format(whence, "%Y-%m");
+---------+------------------------------------------------+
| Month   | Number of Films                                |
+---------+------------------------------------------------+
| 2007-05 | ****                                           |
| 2007-06 | ******                                         |
| 2007-07 | ******                                         |
| 2007-08 | *****                                          |
| 2007-09 | ******                                         |
| 2007-10 | ***                                            |
| 2007-12 | ****                                           |
| 2008-01 | *****                                          |
| 2008-02 | ****                                           |
| 2008-03 | *                                              |
| 2008-04 | *****                                          |
| 2008-05 | *****                                          |
| 2008-06 | ******                                         |
| 2008-07 | ****                                           |
| 2008-08 | *****                                          |
| 2008-09 | *****                                          |
| 2008-10 | ****                                           |
| 2008-11 | *****                                          |
| 2008-12 | ***                                            |
| 2009-01 | ***                                            |
| 2009-03 | **                                             |
| 2009-04 | ***                                            |
| 2009-05 | **                                             |
| 2009-06 | ****                                           |
| 2009-07 | ****                                           |
| 2009-08 | *                                              |
| 2009-09 | **                                             |
| 2009-10 | ****                                           |
| 2009-11 | **                                             |
| 2009-12 | **                                             |
| 2010-01 | **********                                     |
| 2010-02 | **********                                     |
| 2010-03 | *************************                      |
| 2010-04 | *****************************                  |
| 2010-05 | ********************************************** |
| 2010-06 | ***********************                        |
| 2010-07 | ****************                               |
| 2010-08 | **********                                     |
| 2010-09 | ************                                   |
| 2010-10 | **********                                     |
| 2010-11 | ********                                       |
| 2010-12 | *********                                      |
| 2011-01 | *******************                            |
| 2011-02 | *************                                  |
| 2011-03 | **                                             |
| 2011-04 | ************                                   |
| 2011-05 | ********                                       |
| 2011-06 | ***                                            |
| 2011-07 | ****                                           |
| 2011-08 | *********************                          |
| 2011-09 | *                                              |
| 2011-10 | **                                             |
| 2011-11 | ***************                                |
| 2011-12 | ********************                           |
| 2012-01 | ********************                           |
| 2012-02 | *******                                        |
| 2012-03 | *******                                        |
| 2012-04 | *****                                          |
| 2012-05 | ******                                         |
| 2012-06 | *******                                        |
| 2012-07 | **************                                 |
| 2012-08 | ************                                   |
| 2012-09 | ***************                                |
| 2012-10 | *******************                            |
| 2012-11 | ****************                               |
| 2012-12 | *******                                        |
| 2013-01 | *************                                  |
| 2013-02 | *************                                  |
| 2013-03 | ******************                             |
| 2013-04 | ******************                             |
| 2013-05 | ********                                       |
| 2013-06 | ************                                   |
| 2013-07 | **************                                 |
| 2013-08 | ***                                            |
| 2013-09 | *************                                  |
| 2013-10 | ********                                       |
| 2013-11 | ***************                                |
| 2013-12 | ***********************                        |
| 2014-01 | *************************                      |
| 2014-02 | ********                                       |
| 2014-03 | *************                                  |
| 2014-04 | ****************                               |
| 2014-05 | ************                                   |
+---------+------------------------------------------------+
83 rows in set (0.02 sec)