“Mounting” A VDI Image Under Linux

 Computing, Working Notes  No Responses »
Feb 042014
 

So I found myself in the position of wanting to poke around the file system of a virtual Windows machine – the kind of poking around you would prefer I didn’t if it were your Windows machine – and needed to make a VDI disk image available as a block device under Linux so it could be mounted in the normal fashion.

Googling around found some instructions; which didn’t work properly. Solutions were also available, but I’m writing up the ‘fixed’ instructions here to save myself time in case I need it again.

First step is to become root; if you need help doing that, this is probably the wrong page for you!

Next step is to install the Debian package qemu-utils which contains the tool qemu-nbd which we’ll need later. We also need to load the network block device module with a parameter :-

apt-get install qemu-utils
modprobe nbd max_part=16

This parameter is the key here – for some reason the default on at least some of my Linux machines is not to create additional block devices for any additional partitions that show up.

The next step is to ‘attach’ the VDI image (or presumably anything supported by qemu-img which covers pretty much everything popular) and tell the Linux kernel that there may be some new partitions to create device files for :-

qemu-nbd -n -c /dev/nbd0 disk.vdi
partx -a /dev/nbd0
partx: /dev/nbd0: error adding partitions 1-2

(Added the “-n” flag after reading about some more problems and a work-around; as I haven’t tested it, be careful!)

The error from partx indicates that qemu-nbd managed to create the partitions itself, but there are hints that this sometimes doesn’t happen so I’ve included the command here “just in case”. Once the partition block devices are present, they can be used as any ordinary devices.

Once finished, unmount anything mounted and release the block device with :-

qemu-nbd -d /dev/nbd0
rmmod nbd

The New Cold War

 Computing, General, Politics, Security  No Responses »
Jan 272014
 

I’m old enough enough to remember the tail end of the real cold war between the West and the old Soviet Union when we were waving nuclear missiles at each other. And threatening each other with nuclear annihilation.

So it is a bit of an exaggeration to speak of a new cold war when the threat is nowhere near as apocalyptic. But if you take a look at how the old cold war was fought – with espionage, and signals intelligence – you begin to realise we do have a new cold war. Intelligence agencies around the world are cooperating in fighting against a new enemy.

Us.

Oh, they’ll defend themselves by saying that it’s not the normal man or woman in the street they are worried about, but but the terrorists in our midst they are targeting. But to do that they have to spy on us.

They’ll say that they are not spying on the people in their own country; just on those sneaky foreigners. But when GCHQ spies on US citizens, they pass the information they obtain to the NSA; and the NSA passes information on their spying activities to GCHQ.

Which means that what little protection we have against our own intelligence agencies spying on us is effectively meaningless.

How Old Is The Internet?

 Computing, History  No Responses »
Jan 172014
 

There is no clear answer to the question of how old the Internet is. For different definitions of the “Internet”, there will be different starting dates.

For instance, it is commonly held that the pre-cursor to the Internet – ARPANET – could not be called the Internet. And it is true that ARPANET was not the same as today’s Internet even at the lowest possible level. But there is a commonality to ARPANET standards through Internet standards – the very first RFC (issued in 1969) to one of the very latest (RFC7115) are all part of the same body of work.

And whilst the overwhelming majority of ARPANET era standards have been superceeded, there are a few that are still valid today. For example, an early standard for the names of hosts which restricts what characters can be used is still valid and (for example) restricts the names that can be used in email addresses – see RFC608 (it has been updated but the essential restrictions remain).

The next milestone in the history of the Internet came when the older NCP protocol was replaced with TCP/IP in 1982 (actually the “flag day” was 1st January 1983); this immediately raised the possibility of joining networks together and to route between them. Previous to this, different networks had gateway machines which were connected to two (or more) networks. Before the Internet took off, there were more than a few precursor networks – MERIT, JANET, BitNET, …; all of which used different network protocols.

Gateway machines would typically only gateway certain kinds of application traffic from one network to another; typically email was the bare minimum leading to services which would send information via email – at one point you could even “browse” the web using email!

Routing on the other hand allowed end to end communication so it was possible to use applications directly.

The next milestone was allowing commercial traffic on the Internet. The earliest networks were founded for research purposes by the American military or academic organisations, and prohibited commercial traffic Until the core networks allowed commercial traffic we wouldn’t have seen the Internet as we see it today.

There are plenty of other milestones – some would include the foundation of the world wide web (in 1991 and not 1993) as one of the most important. I don’t; simply because it was something that was bound to happen in one way or another.

Linux Dream Cheeky USB Thunder Missile Launcher

 Computing, Linux, Working Notes  No Responses »
Jan 162014
 

This is not original work, but merely a set of notes on how to do the set up. The core information (and the code) came from this blog posting.

Essentially I’ve re-ordered the steps in which to work and excluded anything other than the bare essentials to get it all working. With the intention I can get my missile launcher working at home and at work  😎

Step 1 is to prevent the HID driver from clamping on to the missile launcher. This was done by :-

  1. Editing /etc/default/grub and adding usbhid.quirks=0x2123:0x1010:0x04 to the existing variable GRUB_CMDLINE_LINUX_DEFAULT.
  2. Run update-grub (I always manage to forget this).
  3. Reboot the machine and check /var/log/messages for 2123 (the VendorID) to see if it has been claimed by usbhid (which will show up as a line beginning generic-usb 0003:2123:1010.0006: hiddev0 if it does claim it).

The next step is to download and compile the code given in the blog link above. If you need instructions on how to do this, then you probably need to look elsewhere – it builds easily.

Once built, an sudo insmod launcher_driver.ko will verify that the kernel module loads – you can double check by looking at /var/log/messages.

It’s also necessary to install both the kernel driver and the control program manually :-

  1. Copy the compiled kernel module to /lib/modules: sudo cp  launcher_driver.ko /lib/modules
  2. Edit /etc/rc.local and add the command: /sbin/insmod /lib/modules/launcher_driver.ko
  3. Copy the control program to a sensible location: sudo install launcher_control /opt/bin

There’s probably better ways of doing this, and better places to stick things but as you’re following my instructions you’re stuck with my suggestions! It’s tempting to try a reboot at this stage to verify that this works, but as there’s just one small extra step we may as well get that done too. This is to create a udev rule to set up a device file in /dev.

Create a file (/etc/udev/rules.d/99-usb-launcher.rules) with the following contents :-

KERNEL=="launcher?*",MODE="0660",GROUP="cdrom"

The choice of group name is rather inappropriate except it will work well enough, and I have changed the permissions on this to something a little more restrictive. This can be tested with sudo udevadm trigger which will re-run udev. This should change the permissions on any existing /dev/launcher* file(s). If it doesn’t work, the blog pointer above is the place to head.

Lastly, there’s a couple of corrections to the launcher_control.c that is convenient to make :-

% diff launcher_control.c launcher_control.c.orig
63c63
<         while ((c = getopt(argc, argv, "m:lrudfsht:")) != -1) {
---
>         while ((c = getopt(argc, argv, "mlrudfsht:")) != -1) {
97,98c97
< 		fprintf(stderr, "Couldn't open file: %s\n", dev);
<                 /*perror("Couldn't open file: %m");*/
---
>                 perror("Couldn't open file: %m");

 

It’s All About Reliability, Reliability, and More Reliability

 Computing  No Responses »
Jan 122014
 

Computers have gotten faster … a lot faster. In some cases there is never enough speed, but to a certain extent today’s computers are not noticeably faster than computers of a few years ago. At least not if you do not run benchmarks. So there is little incentive to upgrade that 5 year old desktop machine – unless you are running Windows XP of course (support for which will be dropped soon).

Unless of course you look at aspects other than simple speed – such as reliability.

A few years ago I used to run old Unix workstations in preference to PCs despite their lack of speed, because they were simply more reliable – I could leave a workstation running for weeks without any negative effects. Whereas the PCs I was used to using were just not quite as stable; every so often something unexpected would occur and a reboot would be necessary. Usually at the most irritating possible time.

We expect computers to be reliable, but are all too often disappointed.

Desktop manufacturers may be able to revive the flagging market for desktops by offering something new – desktops with reliability. There are a number of reliability features that are commonly found in servers that could be offered in desktops with only a marginal increase in cost.

Error Correcting Code Memory

Forget the “code” part in the title; without going into a great deal of technical detail, ECC memory automatically corrects memory errors when they occur. And occur they do.

There are a variety of causes of bit errors within memory varying from cosmic rays to atmospheric radiation; the cause does not matter so much. What matters is how frequently they occur. According to small studies and theory, they should be quite rare, but Google have released a paper actually measuring the error rate in a large pool of machines; the error rate is roughly about 5 single bit errors in 8 Gigabytes of RAM per hour.

If true, that’s more than enough to have a significant impact on the reliability of your average desktop PC. If a piece of software has some random instructions changed into something else, it will usually crash or do something strange to your data. Or if that random memory error occurs within your data, then you might expect a strange coloured blob to appear in your favourite photo.

Normal desktop PCs do not come supplied with ECC memory because it is slightly more expensive than ordinary memory. Without going into details, ECC memory uses additional memory to maintain a check on the contents of main memory.

And that costs more. Not a lot more, but in a competitive market, a small saving may lead to increased sales. Of course there are other ways to increase sales – such as by making a feature of ECC memory and reliability.

Storage

We are currently in a transition period between mechanical storage (disks) and electronic mass storage (flash). Flash storage currently offers very fast storage but with a price tag attached meaning it is infeasible for large amounts of storage. That will of course change.

In the meantime we have to deal with two storage solutions; one with a reputation of unreliability (flash) and one that is really unreliable (disks). Both fail with regrettable regularity (although discs will fail more often!) but fail in different ways. Disks themselves are likely to have a short period where they do not work very well before refusing to do anything, although as mechanical devices they can fail in surprising ways too! Flash will tend to fail in a rather nice way – it will get to the point where all attempts to write will fail, but all of the information is still readable.

Because they fail in different ways, we have to cope with their failure in different ways too. Except for the most obvious thing – everything needs to be backed up. And of course getting a backup mechanism up and running is a pretty tedious task.

It would make a great deal of sense for a vendor to offer a cloud-based disaster recovery backup for your system disk(s). An account with a copy of the system disk image is created before your system is shipped. And once on line, your desktop PC sends updates to that image in the cloud. And when the disk fails, you can ask the vendor to ship a replacement disk with almost everything you previously had already put in place.

On a more general note, it is worth mentioning that most consumer hard disks at the bottom end of the market are complete rubbish. And I would pay extra to buy disks from a vendor that :-

  1. Takes ordinary disks and burns them in for a week to verify that they are not going to go bad in the first few months; there’s a NAS vendor (whose name escapes me for the moment) that does this and has one of the lowest disk failure rates on the market despite using relatively cheap and nasty disks.
  2. Ships them in proper packaging that absorbs the shipping bumps and knocks. Just because a disk drive looks intact does not mean it is safe to use.

 And What About The File System?

So far it has all been about the hardware, but there is more we can do about reliability in software too. And carrying on from the previous section, one of those areas is how the operating system stores files on disks.  The software module that does this is (to use the Unix or Linux term) the file system and there are different kinds.

Historically different file systems have assumed that the storage is perfectly reliable. However with the increased awareness of silent data corruption, there are now a few file systems that check for silent data corruption – including what is probably the first: ZFS.

Even if there is a small loss of performance, file systems should detect silent data corruption and correct if possible.

Preparing To Fail

We all know that software is unreliable; to be precise it is not perfectly reliable as it is a great deal more reliable than we give it credit for. After all we only notice the failures; and some of the failures at that.

Rather than trying just to produce reliable software, programmers should be designing software that fails safe without losing any data. See crash-only software.

 Older Entries  Newer Entries