Apr 012024
 

So I was reading 𝕏 and came across one of those memes showing “Chinese bots” making connections to “open” SSH ports to Internet accessible servers. The suggestion to turn off password authentication in favour of public/private key authentication was certainly a sensible suggestion (on a very simplistic level it effectively makes a very strong “password”).

But the “Chinese bots” thing sort of irritated me a bit, so I decided to trawl my personal firewall logs looking for attempts to connect to my ssh port(s). Even ignoring the IPv6 probes, there were 1251 different addresses probing my network (just one public IPv4 address) in the months of March so far.

Why is this irritating? Because the addresses of the machines attempting to break into a non-existent ssh service here are those of compromised machines. They may be in China, or the USA, Russia, etc. but that in no way betrays who is controlling those “bots”.

Anyway, for some data :-

CountCountry
502,US USA 840 United States
128,CN CHN 156 China
97,KR KOR 410 Korea, Republic of
33,SG SGP 702 Singapore
27,BG BGR 100 Bulgaria
26,RU RUS 643 Russian Federation
22,HK HKG 344 Hong Kong
22,GB GBR 826 United Kingdom
20,DE DEU 276 Germany
16,SE SWE 752 Sweden

And “China” isn’t even in the lead in this case! I have included just the top 10 as a long list of random countries with one or two robots isn’t very enlightening.

The key point here is that the national identity of the compromised host attacking tells you nothing about where the true attacker is from. Russia is quite a likely candidate given it’s status as a rogue nation with a known tolerance for cyber criminals (as long as they co-operate with the state when the state needs their skills), but that is just background knowledge.

Jun 082019
 

Quite a while ago, I “borrowed” some inscrutable zsh magic to automatically add the contents of ~/.ssh/known_hosts to a known_hostsi variable and used that variable to perform host name completion for certain commands. Once ssh started hashing the known_hosts file, this broke and I was busy at the time and stopped using it.

Ages later, I’ve fixed it and enhanced it a bit (and arguably made it a bit simpler). Not only does it pick up host names from the known_hosts file but also adds a list from lxc-ls and adds a few static host names (with one exception, not shown). This is done by adding the following to .zshrc :-

knownhosts=( $(sudo -b lxc-ls) )
#       Pick up a list of hosts from lxc-ls
knownhosts+="localhost"
#       Add static hostnames
for x in $(grep -v "|" ~/.ssh/known_hosts | awk '{print $1}' | awk -F, '{printf "%s ", $1}')
do
  knownhosts+=$x
done
#       Pull a list of hosts out of ~/.ssh/known_hosts excluding the Hashed hosts.
zstyle ':completion:*:(ssh|scp|sftp|ping|nmap):*' hosts $knownhosts
#       Commands to use a list of known hosts with

That probably is not the most efficient code, but does have the advantage that it is relatively easy to follow.

One addition is to add the option HashKnownHosts no to ~/.ssh/config.

There is of course a risk associated with disabling the hashing of host names within the known_hosts file. If your host becomes compromised, malicious code can use that file to obtain a list of hosts with which there is a trust relationship making it easier for an attacker to pivot through your network.

Jun 122018
 

This posting is about using the command-line ssh tool for relatively securely copying stuff around, and logging into devices. Many of the tips contained within are things I have had to pry out of the manual page for my own use and these notes are a way of keeping the information around without relying on my brain.

#1: It Comes With Windows

If you are running the latest version of Windows 10, you get the command-line versions of ssh and scp without dropping into the Linux shell :-

Of course you have been able to install ssh clients for Windows for years or even decades, but having it available by default is a big win. Particularly for Windows machines you don’t tweak with your favourite applications.

#2: Public/Private Key Authentication

This the first part of increasing security by only permitting key authentication so that password brute forcing attacks become impossible. With the assistance of an ssh agent (not covered here) or a passphrase-less key pair (not advisable), it is no longer necessary to enter a password.

Of course getting into this sort of thing can be very confusing especially as most instructions tend to get into far too much detail on the cryptography involved. To keep it simple, I shall avoid going on about the cryptography, and concentrate on how to get it to work.

The most important thing to remember about key authentication is that there are two keys – the private key (which should be kept as secure as possible on the client machine) and the public key (which is copied to the devices you want to connect to).

So to get started, you first need to generate a key pair, which can be done with ssh-keygen; this has lots of options, but at this point you can ignore them. After you enter the command, you can simply hit return at all the prompts to generate a key pair :-

Generating public/private rsa key pair.
Enter file in which to save the key (/home/mike/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/mike/.ssh/id_rsa.
Your public key has been saved in /home/mike/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:REMOVED mike@Michelin
The key's randomart image is:
+---[RSA 2048]----+
|=*+o ..  .B*..=o |
|o+++.  . =+o. o+.|
|.BE.+   + .  =  .|
|o+=& . . .    o  |
|. o +   S    .   |
|     .           |
|    SS           |
|              .  |
|     --          |
+----[SHA256]-----+

Of course this is not ideal because there is no passphrase, but to get started with that’s fine. You can ignore most of this output (except for the first item in the following list) but just in case :-

  1. The key pair is saved in the files ~/.ssh/id_rsa (the private key) and ~/.ssh/id_rsa.pub (the public key). The permissions are usually generated properly, but just to be safe you may want to reset the permissions anyway: chmod 0400 ~/.ssh ~/.ssh/id_rsa; chmod u+x ~/.ssh
  2. The key fingerprint can be used to check that when you are connecting that the keys haven’t changed unexpectedly.
  3. Alternatively (and slightly more of a reasonable check) you can check the fingerprint using the “randoart”.

Of course on its own, it doesn’t do much good. You have to copy it into place onto the machine you wish to authenticate to :-

$ ssh username@server mkdir .ssh
$ cat ~/.ssh/id_rsa.pub | ssh username@server cat ">>" .ssh/authorized_keys

Note the quotes around the “>>”; these are significant because you do not want the local machine’s shell to interpret them – they need to be interpreted by the remote machine’s shell. Normally I would simply “scp” the file into place, but appending to a supposedly non-existent file is safer – just in case it does exist and does contain public keys that are currently in use.

There are a whole bunch of options to the command, but the two most important ones are :-

  1. The -t option which is used to specify the key type to generate (dsa, rsa, ecdsa, and ed25519). This is mostly unnecessary, but some older and limited devices do not understand certain key types. And as time goes on, more key types will be declared “insecure”. So you may sometimes find the need to generate more secure keys. The simplest (but not very efficient) process for dealing with such situations is to generate a key for each key type and try each one in turn.
  2. The -f option which is used to specify the output filenames – the private key is saved under the name ‘filename’ and the public key under the name ‘filename.pub’.

#3: SSH Configuration File and Usernames

There are a ton of things that can be done with the ssh configuration file, but for this section I’ll stick with setting the username used to login to specific hosts – not because this is the most interesting thing that can be done, although it is quite useful.

The configuration file can be found (if it has been created) at ~/.ssh/config (with a system-wide version at /etc/ssh/ssh_config). Within that file, you can set global preferences, or host specific preferences :-

Username fred

Host router
  Username admin
Host dns*
  Username fxb
Host ds-* web-*
  Username baileyf
Host *
  Username fred

The first line (Username fred) instructs ssh to use the username ‘fred’ when no username is specified – ssh 192.168.77.98 effectively becomes ssh fred@192.168.77.87.

If you specify the same username within a Host section, the specified username is used for any hosts that the specification following the Host word. In the first case (“Host router”) the username “admin” will be used for any host called “router” but not “router.some.domain”.

In the case of the second clause, a wildcard is used which is very useful for specifying a range of hosts – the example can match “dns01”, “dns01.some.domain”, or even “dns02”. In fact the first Host section is an example of what you should not do – put in a single hostname without a wildcard because it will only activate if the hostname is specified exactly as given. Put a wildcard in there, and it will work whether you use a single hostname or use the fully qualified domain name.

You can also have more than one host specification – as in the “ds-* web-*” list.

And lastly you can (if you choose) use the Host declaration to specify a set of default values – in much the same way that configuration settings in the global context specify default values. Use whatever method you choose.

#4: Cryptographic Incompatibility

I have commented elsewhere on this, but basically the ssh developers have chosen to disable weak encryption by default. Personally I would prefer that ssh throw up huge warnings about weak cryptography, but what is done is done.

If you need to connect to something with weak cryptography, there are three potential ‘fixes’ to allow connections. Each of these is a keyword to add to a specific host section, followed by a specification of what ‘algorithm’ to add.

In each case, a connection attempt will give an indication of what is wrong together with an indication of what algorithm to include :-

» ssh admin@${someswitch}
Unable to negotiate with ${ip} port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

In this case, we can see that it is the KexAlgorithms we need to adjust and the algorithm we need to add is “diffie-hellman-group1-sha1” :-

Host someswitch*
  KexAlgorithms +diffie-hellman-group1-sha1

This can be repeated for Ciphers and (rarely) MACs.

#5: X11 and Port Forwarding

Run X11 gooey programs over an ssh connection? Of course .. why not?

This can be enabled on a host-by-host basis (it is off by default because it can be insecure) using the configuration file :-

Host pica*
  ForwardX11 yes

This is just a special case of port forwarding where a network port is connected (via the ssh session) to a remote network port. Port forwarding can be very useful – for example to access an internal web site temporarily that isn’t (and probably shouldn’t be) exposed with a hole through the firewall.

Of course this can be done with a VPN, but ssh may be simpler :-

Host pica*
  LocalForward 8000 8000

When the connection is made, a local port is opened (tcp/8000) and connected to tcp/8000 on the machine you are logging into.
 

May 042018
 

I had the pleasure of upgrading a server today which involved fixing a number of little niggles; one of which was that connecting to switches suddenly stopped working :-

✗ msm@${server}» ssh admin@${someswitch}
Unable to negotiate with ${ip} port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1

This was relatively easily fixed :-

✗ msm@${server}» ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 admin@${someswitch}
Password: 

Of course doing this command-by-command is a little tedious, so a more permanent solution is to re-enable all the supported key exchange algorithms. The relevant algorithms can be listed with ssh -Q kex, and they can be listed in the server-wide client configuration in /etc/ssh/ssh_config :-

Host *
    KexAlgorithms ${comma-separated-list}

But Why?

According the OpenSSH developers, the latest version of ssh are refusing to use certain key exchange algorithms (and other cryptographic ‘functions’).

Their intention is perfectly reasonable – by default the software refuses to use known weak crypto. I’m fully behind the idea of discouraging the use of weak crypto.

But the effect of disabling weak crypto in the client is unfortunate – all of a sudden people are unable to connect to certain devices. The developers suggest that the best way of fixing the problem is to upgrade the server so that it supports strong cryptography.

I fully agree, but there are problems with that :-

  1. Some of the devices may very well be unsupported with no means to upgrade the ssh dæmon. Now in an ideal world, these devices wouldn’t be on the network, but in the real world there are such devices on the network.
  2. Some devices may not be capable of being upgraded because of processor or memory limitations. Network switches are notorious for having slow processors and tiny amounts of memory, and it is entirely possible that such a device would not be capable of running more exotic and modern crypto. Similarly lights out management processors are often severely limited.
  3. Even if a device is capable of being upgraded, there are the standard problems – the vendor may be slow at releasing updates, change control gets in the way, and lastly resourcing may be an issue – upgrading several hundred switches manually with just one or two people doing it is not going to be a quick job.

Lastly, whilst security is important, breaking things just to make a point is a little extreme. Whilst it is possible to fix the problem, it is something that isn’t immediately obvious to someone who doesn’t routinely configure ssh. And someone, somewhere has had this breakage occur just before they really need to fiddle with a switch Right Now.

There is a far better option available – leave the weak crypto enabled, but warn noisily about its use :-

WARNING!!!!! (2 second delay)
WARNING!!!!! (2 second delay)

The device you are connecting to only supports known weak crypto which means this connection
is subject to interception by an attacker.

You should look at upgrading the device as soon as possible.

Telling people what is wrong noisily and continuing to work is far better than simply breaking with a rather terse message.

Foggy Reflection

 

Dec 222014
 

This is a series of working notes on the Yubikey which is an interesting device used to supplement passwords to make two-factor authentication easier. It is essentially a hardware security token device that pretends to your computer to be a keyboard and enters a one-time only password that can be used to verify your identity – much like a password, but much more secure.

Well perhaps "easier" only if someone does all the configuration for you, although I am inclined to look a bit deeper into such things for my own amusement. My own key is a Yubikey NEO, but much of what follows also applies to the other Yubikey models.

Observations

This is the spot for observations on using the Yubikey over time.

  1. For some reason the Yubikey doesn't always "light up" on my workstation at work. It works fine at home – the green light always turns on ready for a key press – but at work it often seems to flicker and stay out. Not sure what causes this, but it always seems to be persistent when you really need to use it! 

Configuration

… is to some extent unnecessary, but under Linux there are three bits of software that can be installed to configure additional features of the Yubikey :-

  1. The library: https://developers.yubico.com/libykneomgr/
  2. The command-line tool: https://developers.yubico.com/yubikey-personalization/
  3. The GUI: https://developers.yubico.com/yubikey-personalization-gui/

All three build easily from the instructions given. Just make sure to remember to copy the udev rules from yubikey-personalization to /etc/udev/rules.d/ and run udevadm trigger to enable them. This will make sure you can access your yubikey as a console user, so you don't have to become root.

Enabling Linux Authentication

This was all done with a Linux container (LXC), so it could be relatively easily thrown away and restarted. The first step was to install the relevant PAM module :-

# apt-get install libpam-yubico

This pulls in a ton of other required packages.

The next is to grab the unchanging part of your Yubikey token. This is the first 12 characters of what you get when you activate it. Whilst you have it to hand, now would be a good time to create the mapping file – /etc/yubikey-mappings :-

# Yubikey ID mappings
# Format:
#       user-id:yubikey-id:yubikey-id:...
# (But usually only one)
user-id:ccccccsomeid

Next step is to add a little something to one of the pam files. For testing (assuming you have console) access, the relevant file might be /etc/pam.d/sshd but once you have things working, /etc/pam.d/common-auth might be a better choice. Right at the top of the file add :-

auth       sufficient   pam_yubico.so debug id=16 authfile=/etc/yubikey-mappings
#       Added for Yubikey authentication.

Because these things always have problems when you first try them, it makes sense to set up the debugging log :-

touch /var/run/pam-debug.log
chmod a+w /var/run/pam-debug.log

At this point, assuming everything works as expected :-

  1. You will be able to authenticate using ssh using either your Yubikey, or your password.
  2. This assumes your server is able to communicate with the Yubi Cloud.

There are further improvements to be made … and we'll get to those shortly.

But That's Not Two-Factor Authentication!

Indeed not, so we'll fix that right now.

Firstly remove the line we previously added to /etc/pam.d/sshd; because of the way that Debian configures pam, it is less disruptive (i.e. fewer changes) to make the change to /etc/pam.d/common-auth :-

auth       requisite     pam_yubico.so id=16 debug authfile=/etc/yubikey-mappings
#       Yubikey configuration added.
auth    [success=1 default=ignore]      pam_unix.so nullok_secure use_first_pass

But before restarting sshd (you have been doing that haven't you?), you will need to add a Yubikey ID to /etc/yubikey-mappings for the root user.

At this point, you will only be able to authenticate if you enter your username, followed by both your Unix password and activate your Yubikey at the password prompt. Entering both at the same prompt is a little weird especially when you consider that there is no indications anywhere that Yubikey authentication is required.

But we can fix that. First of all, one small change to common-auth – remove the use_first_pass phrase.

Next edit the file /etc/ssh/sshd_config and find the ChallengeResponseAuthentication phrase and set to "Yes" :-

ChallengeResponseAuthentication yes

And after a quick reboot, the log in process works in a sensible way :-

» ssh chagers
Yubikey for `mike': (Press YubiKey)
Password: (Enter Unix password)
Linux chagers 3.14-0.bpo.1-amd64 #1 SMP Debian 3.14.12-1~bpo70+1 (2014-07-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Dec 31 15:37:05 2014
...
</pre>