No ads? Contribute with BitCoins: 16hQid2ddoCwHDWN9NdSnARAfdXc2Shnoa
May 202017
 

I just love messing around with run-time languages that I know relatively little about (and if your sarcasm detector isn’t flashing red about now, take it out and give it a good talking to).

The problem detailed here is something that you are unlikely to encounter unless you get into weird stuff like running an odd-ball window manager, aren’t content with the version of said window manager distributed with your Linux distribution, and are used to re-compiling things from scratch.

It all started when I upgraded Ubuntu on my work machine (to Zesty Zapus). The window manager version was upgraded from 3.5 to 4.0, which broke on my configuration file (3.5); not a big problem I thought, as I had already upgraded my window manager at home to 4.1 and reconfigured the configuration file. I copied the updated configuration file from home into place.

And it failed. Apparently I use 4.1-isms within the file. As I was not happy about tinkering with the file to downgrade it (in a language I know relatively little about), I decided to re-compile Awesome 4.1 instead.

Which failed with a weird error :-

» awesome --version
awesome v4.1 (Technologic)
 • Compiled against Lua 5.3.3 (running with Lua 5.3)
 • D-Bus support: ✔
 • execinfo support: ✔
 • xcb-randr version: 1.4
 • LGI version: [string "return require('lgi.version')"]:1: module 'lgi.version' not found:
	no field package.preload['lgi.version']
	no file '/usr/local/share/lua/5.3/lgi/version.lua'
	no file '/usr/local/share/lua/5.2/lgi/version.lua'
	no file '/usr/local/share/lua/5.3/lgi/version/init.lua'
	no file '/usr/local/share/lua/5.2/lgi/version/init.lua'
	no file '/usr/local/lib/lua/5.3/lgi/version.lua'
	no file '/usr/local/lib/lua/5.3/lgi/version/init.lua'
	no file '/usr/share/lua/5.3/lgi/version.lua'
	no file '/usr/share/lua/5.3/lgi/version/init.lua'
	no file './lgi/version.lua'
	no file './lgi/version/init.lua'
	no file '/usr/local/lib/lua/5.3/lgi/version.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.3/lgi/version.so'
	no file '/usr/lib/lua/5.3/lgi/version.so'
	no file '/usr/local/lib/lua/5.3/loadall.so'
	no file './lgi/version.so'
	no file '/usr/local/lib/lua/5.3/lgi.so'
	no file '/usr/lib/x86_64-linux-gnu/lua/5.3/lgi.so'
	no file '/usr/lib/lua/5.3/lgi.so'
	no file '/usr/local/lib/lua/5.3/loadall.so'
	no file './lgi.so'

Which had me stumped for a while, and it turns out that DuckDuckGo didn’t have an obvious fix (one of the reasons for writing this).

Eventually I figured out that awesome was not finding the LGI module (I can be slow at times) which was odd because it was definitely installed. However it turns out that it was installed in /usr/share/lua/5.2/lgi. So despite having lua 5.3 installed, extra lua modules can only be seen if you have lua 5.2 installed?

The “fix” for this was to create an environment variable telling LUA to search for files in rather more places before starting Awesome :-

export LUA_PATH="/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.2/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/share/lua/5.2/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;/usr/share/lua/5.3/?.lua;/usr/share/lua/5.2/?.lua;/usr/share/lua/5.3/?/init.lua;/usr/share/lua/5.2/?/init.lua;./?.lua;./?/init.lua"

This was created by running lua from the command line and running print(package.path) to display the default setting, and adding the 5.2 equivalent for many elements.

As to whether it works or not, well I cannot be sure (I’m not going into work on a weekend just to check if the window manager fires up), but Awesome itself seems happy with the result :-

» awesome --version
awesome v4.1 (Technologic)
 • Compiled against Lua 5.3.3 (running with Lua 5.3)
 • D-Bus support: ✔
 • execinfo support: ✔
 • xcb-randr version: 1.4
 • LGI version: 0.9.1

So it can find LGI, but whether it can do anything useful with it remains to be seen!

Feb 122017
 

A very long time ago, I used to collect spam in order to graph how much spam a single mail server was likely to get over time, and almost as long ago, I lost interest in maintaining it. As a consequence I still get a ton of spam every day and after a long period of procrastination I have been slowly raising defences against spam.

This particular recipe is not really a defence against spam – it verifies that the remote server is properly DNS registered with a reverse DNS registration – in other words that the IP address it is connecting from is registered. This is a requirement for all mail servers, and as it turns out, spammers don’t care for registering their servers in the DNS.

This ACL snippet goes into the ACL for checking the recipient or for checking the message :-

 deny
   message = Your mail server is not properly DNS registered
   log_message = BLOCKED: No rDNS
   condition = ${if eq{$host_lookup_failed} {1} {1}{0}}
   # Check rDNS and block if not registered

There are three items of interest :-

  1. The message is intended to be easily read by recipients to determine what the problem is. It turns out that many people do not read NDRs, but if we get the message right at least we are doing the right thing.
  2. The log_message is intended to make automating log parsing easier.
  3. Within the condition, the $host_lookup_failed variable indicates that the reverse DNS lookup returned NXDOMAIN and not that it timed out (which would be $host_lookup_deferred).

That’s all there is to this little piece of configuration.

Jan 192017
 

Entropy.

Any serious cryptographic routines needs a good source of random numbers, and whilst Linux provides a random number generator by default it’s sources of entropy can be somewhat limited. Especially when you’re talking about a virtual machine.

Indeed if you try to pull too much randomness out of the Linux entropy pool (especially when it is especially limited), what you get might not be quite as random as you expect.

Which is where hardware randomness generators come in. And I finally have one (actually two), and have hooked them up. You may be able to guess what time I plugged it in from the graph below :-

So what real world difference does it make?

Well nothing is dramatically obvious, but :-

  1. I have slightly more confidence that any cryptographic software I might run has a good source of randomness and is less likely to accidentally perform poorly (in terms of cryptographic strength).
  2. Some cryptographic software blocks if the Linux entropy pool is empty; with a hardware source I can be more confident that any performance issues are not due to a lack of randomness.
Dec 112016
 

Vi (or vim) is one of those editors that at first appearance appears to be insanely user-hostile, and some will say it looks the same way on a second, and third glance. Yet it remains one of the most popular editors under Linux, and even if you choose another editor as your mainstay, you are likely to encounter it in use a fair bit.

(Apologies for the little line at the top of that screenshot; lazy editing)

The strange thing about vi is that nobody uses the real thing any more (well almost nobody), but instead clones of one kind or another. That is mostly because vi originated in the commercial world of Unix, and clones were written to be open-source. It is perhaps worth remembering that vi has remained the mainstay of editing under Unix-based operating systems for three decades despite there being many alternatives.

It may look plain, but vi has almost every editor feature you can think of and almost certainly a few you never thought of.

The Modes

Almost every every other editor currently in use is a modeless editor, or at least mostly modeless. Vi is different in that it has three different modes that operate differently – insert mode, command mode, and ex-mode (essentially for extended commands). Of the three modes, the insert mode is the most like other editors, although are relatively few commands to use. As you can see in the screenshot above, the words “- INSERT -” appears at the bottom of the screen whenever you are insert mode (and the same for replace mode which is effectively the same).

Most commands are performed in the command mode, which can be thought of as the default mode – there is nothing saying that you are in command mode. If in doubt, you can press Esc to get from insert mode into command mode. There are some who will argue that in fact vi is modeless and that the “insert mode” is in fact a parameter to the insert command. This has a certain ring of validity to it – if you enter 32iHelloEsc in command mode, you will end up with 32 copies of the word “Hello” inserted.

But conventionally vi is written of as a mode-based editor, so it is best to think of it as such until you have learned enough to throw off conventional wisdom and go your own way.

The last mode is ex-mode, which at a basic level is covered only in enough detail to tell you how to get out of it! It is entered from command mode with the “:” command, at which point the cursor moves to the end of the screen leaving you free to type in lengthier commands. To exit simply hit the backspace key until the cursor returns to its normal location.

The Insert Mode

(and the replace mode)

The insert mode is started with a variety of commands, but the simplest is i(nsert).

Once in insert mode, you can start typing normal text without worrying about what commands it will run. There are a fair few things you can do with the control keys, but we’ll skip over those for the “basics”. To correct a few historical limitations :-

  1. You can move the cursor around with the arrow keys. It might seem a bit strange to say so, but the original vi didn’t allow you to move in insert mode partially because it pre-dates arrow keys (yes, really!) and had to use commands to move the cursor around the screen.
  2. You can move anywhere within the file and make changes anywhere; not just where the original change was intended. This may seem like an unnecessary feature to explain, but when you are changing a single word, it can seem wrong to also go somewhere else in the file and make changes elsewhere.

Without going into the more esoteric features, there is not a great deal more to say about the insert mode except it is exited with the Esc key.

The Command Mode

The movement commands :-

Arrow Keys Moves the cursor
h & l Moves the cursor left and right.
j & k Moves the cursor down and up.

The use of the h,j,k, and l keys to move the cursor around the screen seems rather bizarre except when you realise that some early terminals connected to Unix systems lacked cursor keys. They remain for compatibility reasons and because some people feel that they can be quicker to use as they require less hand movement than the cursor keys, or like me that those keys are burnt into muscle memory and so they are used almost without thought.

b(ack) Move backwards one word.
e(nd) Move forward to the end of the word.
f(orward){char} Move forwards on the same line to the next occurrence of {char}.
n(ext) Move to the next occurrence of the last search.
 / Search for something.

The most basic command for deleting text is “x” which deletes the character under the cursor, but a hint of what can be done with vi comes with the d(elete) command. The d(elete) command takes a movement as a parameter, and deletes from the current cursor position until where there movement takes you :-

dd Deletes line.
d$ Deletes to the end of the line.
d0 Deletes until the beginning of the line.
dw Delete until the end of the word.
diw Delete “in” word – deletes the current word.
df{char} Deletes until the next occurrence of {char}

But we now move finally to adding text :-

i(nsert) Insert at the current cursor.
o(pen) Open a new line below the current line.
O(pen) Open a new line above the current line.
a(ppend) Append text after the cursor.
A(ppend) Append new text at the end of the line.

Lastly, we can save and exit vi with “ZZ”.

Ex-Mode

This is going to be even more truncated than the last section (I know the last section doesn’t seem truncated, but trust me – it is!). There is a great deal more to this mode than just the three commands below :-

:write (or :w)

Writes the file being edited. Two options I am going to mention here.

Firstly you can add a filename to the command to write to an alternate file (:write new-filename) – very handy if you find you’re making changes to a file that you do not have permission to overwrite.

If you need to override a warning vi has about overwriting the current file, you can do so by appending an exclamation (!). Just don’t do it automatically (I’ve a sorry song to sing about doing that!).

:quit

And to quit vi, simply use the :quit command. If there are unsaved changes in the file you are editing, it will stop you, in which case if you really want to lose your changes add an exclamation (!).

Sep 012016
 

Although I use graphical on-screen calculators for many calculations, it can sometimes be convenient to perform calculations at the command-line (or in shell scripts). In which case the old tool is expr :-

» expr 3 \* 9 
27

Very convenient; even though I can do such a calculation in my head there are circumstances where checking with a calculator is suitably cautious. You can of course perform calculations directly in the shell; if you are using a modern shell such as zsh or bash :-

» echo $((3 * 9))
27

Whilst convenient, such methods do have their disadvantages :-

  • The expr tool takes it’s expression after the shell has had it’s way with interpreting it – which is why I have escaped the “*” to multiply. You cannot put quotes around the expression either as expr assumes it to be a string.
  • These calculations are integer calculations, so you cannot find out what 77/4 is (19.25). Oops! Turns out that if you make one of the numbers in the expression a float, then the result is properly calculated: echo $((77.0/4) -> 19.25.
  • These calculators are limited to relatively small numbers – according to zsh, 2^63 is -9223372036854775808

If you need something a little more sophisticated then qalc (this is the command-line interface for Qalculate!) makes a pretty good command line calculator. It has to be installed with sudo apt-get install qalc and once installed it should be run interactively to get the initial configuration out of the way :-

» qalc
You need the download exchange rates to be able to convert between different currencies.
You can later get current exchange rates with the "exchange rates" command.
Do you want to fetch exchange rates now from the Internet (default yes)? yes
> quit

Once installed you can perform calculations in the same way as expr (although you can enclose an expression in quotes) :-

» qalc "3 * 9"
3 * 9 = 27
» qalc "2 ^ 72"
2^72 = approx. 4.7223665E21
» qalc "0xff"  
255 = 255
» qalc "86400s to hours"                                                                  
86400 * second = 24 h

You can add the “-t” option to prevent qalc telling you the expression it calculated; perhaps more useful in scripts than interactively.

damascus-unix-prompt

Sep 012016
 

One of the advantages that ZFS brings, is that it is so easy to create file systems, that you can create them for purposes that you would not previously do. For example, I have an additional file system mounted under my home directory for a certain application that generates a lot of data that I do not need backed up. Because the script I use to back up stuff does not cross file system boundaries (i.e. it does not descend into a directory that contains a mounted file system), I can simply exclude a large amount of frequently changing data by making a file system.

Or I might (as it happens I do not, but I could well do) create file systems for large lumps of data to easily see how much space they occupy – perhaps ~/Pictures. You can run a command like du -sh ~/Pictures, but that is an expensive command (it takes a while) and it tells you how large the files are; not how much space they occupy on disk. And on-disk compression can make that a significant difference! So simply run df -h ~/Pictures if that directory is on a separate file system.

But there is a bit of a gotcha with that. If you create such file systems in the normal way (such as zfs create pool/mikes-pictures; zfs set mountpoint=/home/mike/Pictures pool/mikes-pictures) you risk creating a situation that may prevent your home directory from mounting. If the “child” file system is mounted before the parent, it will not be possible for the parent file system to be mounted when booting.

Instead create the hierarchy properly :-

zfs create pool/h2
mkdir /h2
zfs set mountpoint=/h2 pool/h2
zfs create pool/h2/mike
zfs create pool/h2/mike/Pictures
ls /h2/mike/Pictures

You will also have to fix the permissions, but this is a far safer way of organising things suitable for future file system creation.

damascus-unix-prompt

Aug 292016
 

It seems that occasionally GNOME can go a little screwy and its fancy mouse pointer plugin can result in an invisible mouse pointer. Which makes doing anything just a little bit tricky.

If you can open a terminal, enter the command :-

gsettings set org.gnome.settings-daemon.plugins.cursor active false

And all should be well. At least until it decides to turn itself back on again (so make a note of this fix!).

damascus-unix-prompt

Mar 252016
 

Recently I have been seeing quite a lot of usage of random.org (to pick out winners of various kinds of competitions; and no I’m not a winner). The documentation on that site are reasonable with regard to pseudo-random number generators but are not quite correct with regard to the source of random numbers under Linux. And for non-cryptographic uses, the following is fine.

The use of random.org momentarily made me wonder how I would do the equivalent at the Unix (or Linux) command-line, and having used the command before, the shuf command came to mind. To be honest shuffling is not what I think of randomisation given how bad I am at shuffling cards, but despite the name, shuf does pretty well at randomising things :-

» seq 1 10 | shuf
4
5
8
7
2
1
10
9
6
3

The seq command generates a sequence from 1-10 as given. It turns out that shuf can do it itself :-

» shuf -i 1-10
7
3
5
6
9
10
8
1
4
2

The most common (relatively) use I have for shuf is to pick out a random line or two from a file. By using the -n option, shuf can do this. The following example makes use of an example file which contains a small number of first names :-

» shuf -n 1 first-names 
Julian
» shuf -n 1 first-names
Ian
» shuf -n 1 first-names
Craig

If you have just a small selection to make, you can provide the list on the command line with the -em option :-

» shuf -n 1 -e Male Female
Female

And that is pretty much all there is to it – a simple tool that does just one thing well.

damascus-unix-prompt

Mar 052016
 

Just to amuse myself, I’ve been re-reading and re-learning the Unix shell’s ${} detailsand it occurred to me that whilst these were all very well and cute, they very easily lead to impenetrable code. But they are more efficient.

Take the following two ways of getting the current date :-

✓ mike@pica» print -P "%D" 
16-03-05
✓ mike@pica» echo $(date) 
Sat 5 Mar 13:14:38 GMT 2016

It’s not exactly helpful that they return the date/time in different formats. But glossing over that for the moment, which one is clearer? That is right – the second one clearly says that it is going to “echo” the date. Even if this usage is particularly stupid (as date will echo the date all by itself), the second wins as far as clarity goes.

However it is also less efficient – rather than get the date and show it to the terminal, the shell invokes a sub-process to display the date, captures it and then uses it to show to the terminal. In the old days when terminals consisted of printing mechanisms that actually hit a template of a letter against an inked up ribbon against a roll of paper and hoped that the result was readable, this inefficiency could result in very slow code.

But today this level of inefficiency should not make that much difference, and if it does, then why are you writing code in the shell? There are far better languages out there.

In addition, there is a bit of a gotcha with the print -P “%D” option … it only works if you happen to be using zsh :-

✓ mike@pica» print -P "%D"
16-03-05
✓ mike@pica» /bin/sh
$ print -P "%D"
file: option requires an argument -- 'P'
Usage: file [-bcEhikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
            [-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
       file -C [-m magicfiles]
       file [--help]
Warning: unknown mime-type for "-P" -- using "application/octet-stream"
Error: no such file "-P"
Error: no such file "%D"
$ 
✗ mike@pica» /bin/ksh
$ print -P "%D"
%D
$ 
✓ mike@pica» /bin/bash
mike@pica:~/.lyx$ print -P "%D"
file: option requires an argument -- 'P'
Usage: file [-bcEhikLlNnprsvz0] [--apple] [--mime-encoding] [--mime-type]
            [-e testname] [-F separator] [-f namefile] [-m magicfiles] file ...
       file -C [-m magicfiles]
       file [--help]
Warning: unknown mime-type for "-P" -- using "application/octet-stream"
Error: no such file "-P"
Error: no such file "%D"
mike@pica:~/.lyx$ exit

Confusing is it not?damascus-unix-prompt

Of course if the shell would intercept common usages such as $(date) and optimise them, that would be perfectly reasonable.

Jan 052016
 

A bit of a simple one this … if you are looking at converting an Intel hex format file that looks like the following :-

2016-01-05_2123

Then it is relatively trivial under Linux (Debian). The relevant tool is probably installed anyway; unless you are not compiling software which may be a marginal activity for weird people but so is converting ihex files. But just in case, you can install it with: sudo apt-get install binutils.

Once installed (or being already present) the conversion process is as simple as :-

» objcopy -I ihex -O binary somefile.hex somefile.bin

Be careful to specify the second file name or objcopy will overwrite the original hex file (don’t ask how I discovered this!).

By continuing to use the site, you agree to the use of cookies. more information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close