Dec 082018
 

I recently bought a second-hand camera – but this is not specific to photography (but perhaps particularly relevant). The seller threw in an old SD card which was nice of them (although unnecessary for me).

After doing the photo thing with the new-to-me camera, and having carefully replaced the SD card, it occurred to me that I could test a file recovery tool to see if there was any previously shot photos on the card.

Using photorec, I fired it off and came back 30m later – not because it’s particularly slow but I have spent far too much time watching the equivalent of a progress bar, and I would rather get on and do something useful.

By the time I came back, it had recovered in excess of 1,000 images and videos. It turns out to be probably the most boring collection of photos you can imagine – an ordinary collection of family (not your own) photos would be interesting in comparison.

I won’t be including any of those recovered photos here because that would be unprofessional and potentially embarrassing to the camera seller (although they would most likely never find out). 

But you can easily imagine how such a recovery could be potentially embarrassing; even distressing. We usually choose whether a photo should be made public or not.

So how do you protect such things from happening? Is it sufficient to format a card in camera?

No it isn’t. Tools such as photorec are designed to recover images from cards where the images have been deleted or when the card has been formatted. Surprisingly enough, formatting a card does not overwrite all of the data blocks on a storage device; it merely replaces the data structures that allows an operating system to find files with a new blank structure.

So what are the solutions to keep your private photos to yourself?

It should be emphasised that this is advice intended to protect you from personal embarrassment; if there are legal or risk to life issues involved, seek professional advice.

The first rather obvious solution is to never give away or sell old cards; if you want to dispose of the cards, destroy them. It is not as if you could recover much by selling them – who wants a 5-year old 512Mbyte SD card?

If you do want to let others use your old cards, then use a special utility to destroy the contents completely; optionally (but nice for the recipient) is to then format the cards afterwards.

If you are using Windows (or macOS although the following Linux recipe can be adapted), then you will need a tool such as SafeWiper. There are those who claim that Windows format can do the job, but I wouldn’t trust it – the “quick format” option is the default which definitely doesn’t erase the data from the disk, and I have not personally checked that a “slow format” really removes the data beyond recovery with normal tools.

Whatever method you choose, check, double-check, and triple-check that the device you are erasing really

The first step under Linux is to identify the block device path to erase. You may well find that your SD card is automatically mounted when you plug it in. So running df from the command-line will give you a device path (/dev/sdb

But to double check, run lsblk

✓ mike@Michelin» lsblk -o NAME,FSTYPE,MOUNTPOINT,VENDOR,MODEL,SIZE | grep -v loop 
NAME                    FSTYPE      MOUNTPOINT                      VENDOR   MODEL              SIZE
sda                                                                 ATA      SAMSUNG MZNTY128 119.2G
├─sda1                  vfat        /boot/efi                                                   512M
├─sda2                  ext4        /boot                                                       732M
└─sda3                  crypto_LUKS                                                             118G
  └─sda3_crypt          LVM2_member                                                             118G
    ├─ubuntu--vg-root   ext4        /                                                         114.1G
    └─ubuntu--vg-swap_1 swap        [SWAP]                                                      3.9G
sdb                                                                 Generic  USB  SD Reader     3.8G
└─sdb1                  vfat        /media/mike/disk                                            3.8G

Note that how we have “USB SD Reader” alongside /dev/sdb and that it’s size is just 4Gbytes. So we have three confirmations that this is the device we want to erase.

To erase it, first we unmount it, run a hdparm command to erase it, and erase it a second time :-

✓ mike@Michelin» umount /dev/sdb1
✓ mike@Michelin» sudo hdparm --security-erase NULL /dev/sdb
security_password: ""

/dev/sdb:
 Issuing SECURITY_ERASE command, password="", user=user
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
✓ mike@Michelin» sudo dd if=/dev/zero of=/dev/sdb bs=64M

Whilst we’re waiting for the “dd” command to finish writing zeros all over the SD card, why are we erasing this twice? 

We’re using hdparm

And I then suggest using the old slow method of “dd” as well because there is nothing wrong with being cautious in this area.

Misty Trees
Nov 112018
 

If you use the Unix or Linux command-line, you may very well wonder about the origins of some of the “special” characters. One of those is tilde (~) which is expanded by the shell into “home” :-

✓ mike@Michelin» echo $HOME                        
/home/mike
✓ mike@Michelin» echo ~
/home/mike
✓ mike@Michelin» echo ~root
/root

This doesn’t of course work in general; just in the shell.

But where did this usage originate?

As it turns out, it was the markings on the keyboard of the ADM3A terminal :-

If you used Unix in the late 1970s/1980s, you may very well have used the ADM3A terminal and it seems that those who added the tilde feature to the Unix shell were amongst the users.

Oct 092018
 

And yes that includes me

For those who don’t know, RFC1918 is the Internet standard that allocates the addresses used for private networks – 10/8, 192.168/16, or 172.16/12. 

And in reference, specifically :-

To minimize the risk it is strongly recommended that an organization using private IP addresses choose randomly from the reserved pool of private addresses

(Apologies for the incorrect spelling above; it’s a quote from an American source)

This was reinforced to me yesterday whilst I was working from home, and I had trouble with a site-to-site VPN joining my network to work’s. As it happens there was no addressing conflict, but I had to install many routes more than should be necessary.

And I keep seeing this sort of thing; joining multiple networks when everybody is using 10/8 is a continual game of chicken – when are we going to find ourselves in conflict? 

Of course there is a “fix” for this – NAT. The real fix of course is to use global IPv6 addresses even for devices and networks that will never be on the global Internet.

The Window
Oct 032018
 

I have a Python script that over-simplifying, reads very large log files and runs a whole bunch of regular expressions on each line. As it had started running inconveniently slowly, I had a look at improving the performance.

The conventional wisdom is that if you are reading a file (or standard input), then the simplest method is probably almost always the fastest :-

for line in logstream:
    processline(line)

But being stubborn, I looked at possible improvements and came up with :-

from itertools import islice
    
while True:
    buffer = list(islice(logstream, islicecount))
    if buffer != []:
        for line in buffer:
             processline(line)
    else:
        break

This code has been updated twice because the first version added a splat to the output and the second version (which was far more elegant) didn’t work. The final version 

This I benchmarked as being nearly 5% quicker – not bad, but nowhere near enough for my purposes.

The next step was to improve the regular expressions – I read somewhere that .* can be expensive and that [^\s]* was far quicker and often gave the same result. I replaced a number of .* occurrences in the “patterns” file and re-ran the benchmark to find (in a case with lots of regular expressions) the time had dropped nearly 25%.

The last step was to install nuitka to compile the Python script into a binary executable. This showed a further 25% drop – a script that started the day taking 15 minutes to run through one particular run ended the day taking just under 8 minutes.

The funny thing is that the optimisation that took the longest and had the biggest effect on the code showed the smallest improvement!

Four Posts
Sep 102018
 

If you have not heard, Steam have added a compatibility layer to Steam which allows a limited number of Windows games to run. The “compatibility layer” is in fact a fork of WINE called Proton.

Peered at from 500 metres away, Proton allows Windows software to run (or not infrequently crash and burn) by translating the Win32 API into Linux APIs, and translating the variety of graphics APIs into Vulkan. That is a really difficult thing to do.

I have taken a very quick look at the new Steam client (and “Proton” is no longer part of a beta release of the Steam client – it’s in the standard client). It works perfectly adequately, although you will have variable experiences running Windows software.

For some reason this news has captured the imagination of a number of ‘tubers who are more gamers than Linux users, which has lead to some misunderstanding :-

  1. This is not Linux gaming; it is Windows gaming under Linux. If you have a bad experience with Steam under Linux, you are not experiencing a bad time with Linux gaming. Linux gaming involves native Linux software, and yes there is some out there.
  2. Problems with Steam could well be down to the Proton compatibility layer with unsupported API calls or buggy usage of the Win32 API which relies on Windows behaving in a certain way for undefined parameters.
  3. In addition problems with Steam could be due to the hardware you are running; take a game that works perfectly fine with an Nvidia card. It may behave problematically with an AMD card or even a different Nvidia card. Or the other way around.

The important thing to remember when looking at videos about Steam is that the person looking at Steam may not be the most experienced Linux user out there. That is not necessarily bad – the whole purpose of Steam is to be able to run games easily without a whole lot of Linux experience.

But they may not be understanding properly what is going on – for example the first thing I would do as a professional game-orientated ‘tuber would be to try out a selection of games with an nvidia card, and then repeat using an AMD card – just to see if things work better, worse, or at least differently.

And again, this is not about Linux gaming but about allowing easy access to old Windows titles that someone may have bought in the past. 

Pentland Hills