Well, this is kind of a weird one. But most of the systems I run are Linux-based, and over the years I have ‘developed’ a simple script that I run from my main workstation which iterates through all of the systems applying updates.
As non-interactively as possible – it could even be scheduled to run automatically (although I don’t for no good reason).
But it had one great weakness – it didn’t update my Windows 11 virtual machine. Which wasn’t a serious problem because Windows could and did update itself. But it did result in software installed with winget getting left behind.
So I sorted it …
Install OpenSSH server on Windows: PS: Add-WindowsCapability -Online -Name OpenSSH.Server (this might need the version number which is best obtained using Get-WindowsCapability -Online | Where-Object Name -like ‘OpenSSH*’.
Copy your chosen ssh authentication public key into c:\users\${username}\.ssh\authorized_keys file.
Configure c:\programdata\ssh\sshd_config to permit public key authentication (“PubkeyAuthentication yes”).
Whilst in the same file, comment out the section with the line beginning “Match Group administrators” which whilst makes things less secure did at least work! The section does refer to a file: c:/ProgramData/ssh/administrators_authorized_keys but adding to this file didn’t seem to work for me.
Verify that the daemon is running: PS: get-service -name sshd
If it shows as not running, enable with: PS: set-service -name sshd -startuptype ‘automatic’
And either reboot, or start it manually: PS: start-service -name sshd
At this point you should be able to login with a simple ssh username@hostname command. If not you’ve either left something out, or I have!
At this point you should be able to run the relevant update commands :-
ssh username@hostname UsoClient ScanInstallWait. Operating system updates which may or may not work, so I wouldn’t disable the automatic updates at this point.
ssh username@hostname winget upgrade –all. This updates additional software (something I’ve called “layered products” in the past) installed via winget (or the Microsoft “Store”. This can sometimes stop with a mysterious error but should usually work.
So I decided to upgrade the firmware on my ASRock TRX50 WS motherboard tonight. Partially because I had planned on trying it to sort out a mysterious crashing problem (which turned out to be the world’s worst SATA SSD ‘error’), and partially because I’d like to make sure I know how the process works. And funnily enough, finding ASRock’s instructions aren’t so simple.
The first really rather obvious step is to download the firmware from the ASRock support site. This comes down as a ZIP file, which needs to be unpacked :-
TRX50-WS_9.03.ROM
This needs to be copied to a USB stick formatted as FAT32, but whilst you’re checking that make sure that the partition type is set to an appropriate value (0x0b is the value I used; the second time), because it turns out that the ASRock firmware won’t recognise a FAT32 filesystem just based on the actual filesystem – it checks the partition types.
But before you shut down and start the upgrade process, record any firmware settings you may have made … for better or worse, the upgrade will reset any changes you have made.
Starting the upgrade is fairly simple – go into Setup, move across to Tools and select the “Instant Flash” option. This will pop up a menu of different firmware version files it has found that are compatible with your motherboard. Select the version you want (in my case it was just one option), and press Return.
After a warning, it’ll start the upgrade process; this consists of :-
A progress bar which slowly progresses to 100%
A reboot which takes you back into the firmware.
A second progress bar which also progresses slowly.
At some point when this has finished, it’ll just sit there for a few minutes and finally start booting with the new firmware.
Of course in my case, the settings reverting to default values resulted in the SlimSAS controllers both being reset to “NVME” rather than “SATA” meaning half my storage array wasn’t present! But it all worked in the end :-
✓ root@pica» dmidecode -s bios-version
9.03
Of course ASRock claim you only do a “BIOS Upgrade” (I hate that word “BIOS” – it’s not really appropriate) when it is absolutely necessary, but an upgrade when it isn’t necessary isn’t a bad idea. Just to get practice.
It should be noted that the firmware should be update-able with fwupdmgr so any urgent updates may well come via that route.
So I was reading 𝕏 and came across one of those memes showing “Chinese bots” making connections to “open” SSH ports to Internet accessible servers. The suggestion to turn off password authentication in favour of public/private key authentication was certainly a sensible suggestion (on a very simplistic level it effectively makes a very strong “password”).
But the “Chinese bots” thing sort of irritated me a bit, so I decided to trawl my personal firewall logs looking for attempts to connect to my ssh port(s). Even ignoring the IPv6 probes, there were 1251 different addresses probing my network (just one public IPv4 address) in the months of March so far.
Why is this irritating? Because the addresses of the machines attempting to break into a non-existent ssh service here are those of compromised machines. They may be in China, or the USA, Russia, etc. but that in no way betrays who is controlling those “bots”.
Anyway, for some data :-
Count
Country
502,
US USA 840 United States
128,
CN CHN 156 China
97,
KR KOR 410 Korea, Republic of
33,
SG SGP 702 Singapore
27,
BG BGR 100 Bulgaria
26,
RU RUS 643 Russian Federation
22,
HK HKG 344 Hong Kong
22,
GB GBR 826 United Kingdom
20,
DE DEU 276 Germany
16,
SE SWE 752 Sweden
And “China” isn’t even in the lead in this case! I have included just the top 10 as a long list of random countries with one or two robots isn’t very enlightening.
The key point here is that the national identity of the compromised host attacking tells you nothing about where the true attacker is from. Russia is quite a likely candidate given it’s status as a rogue nation with a known tolerance for cyber criminals (as long as they co-operate with the state when the state needs their skills), but that is just background knowledge.
This is a collection of notes from my upgrade to an ASRock TRX50 WS motherboard fitted with an AMD Threadripper 7970X processor (32 cores) and 256Gbytes of memory. The upgrade meant that I retained the case, drives, graphics card, etc. from the previous system.
Most of the problems encountered were due to user stupidity.
First of all, whilst many of us have heard about the amount of time that DDR5 takes to “calibrate” itself, what I didn’t know was that the firmware status code shows “00” during this process (a dedicated “I’m messing with memory” code would be handy). And whilst it takes a while to do, if it takes longer than about 5m, then something else is wrong.
In my case it turned out that I hadn’t read the instructions properly and I hadn’t connected enough power connectors. To get it to work, I needed the usual 24-pin power connector, an 8-pin connector, and a 6-pin connector all connected on the “drive” side of the motherboard (opposite the side with the PCIe slots). Once that was sorted, the system was up and running.
The remaining notes relate to “tweaking”.
Booting Linux
Of course I use Linux, what the hell else would I use? FreeBSD? Well, that would be a good choice.
The biggest problem I had booting Linux was changing the netplan configuration to pick up the new network interfaces. In my case, the Marvell interface (the 10G one) came up as enp65s0 and the Realtek interface (the 2.5G one) as enp69s0. Because I’m bound to plug the cable into the wrong interface, I simply bonded the two interfaces together; the relevant section of my netplan configuration is as follows :-
Yes, you can choose silly names here. And yes the bonding works fine – just now I swapped the cable over to the “right” NIC with numerous active network connections, and everything stayed alive.
Firmware Upgrade
The motherboard was supplied with version 6.04 of the firmware (I refuse to call this a “BIOS” because it just isn’t “basic” any more) whereas the latest was 7.09. The process is fairly simple :-
Save it to a FAT32 USB disk – I used a vfat formatted disk and I have a sneaking suspicion that exFAT will work too. The “Instant Flash” instructions by ASRock are obviously somewhat dated – it even mentions that saving to a floppy disk will work!
Reboot the system and start the UEFI firmware. Select “Tools” and “Instant Flash”.
Follow the on-screen instructions.
If you’re replacing a motherboard you won’t need detailed instructions here, but it is worth mentioning that the process takes a couple of reboots, and the second involves doing that memory calibration thing, so it takes an unusually long time to start.
I didn’t go to the effort to time the whole process, but my system went down at 18:04 and was back up at 18:15. So roughly 10 minutes.
SlimSAS
This isn’t currently 100% confident as I haven’t plugged anything in yet (ignoring a failed attempt when I assumed it work just work), but the SlimSAS ports can be configured for SATA mode in the firmware. Just go to Advanced, Chipset, go to the end of the list (which involves scrolling) past the settings for the PCIe slot configuration parameters and set :-
SLIMSAS1 Mode: SATA
SLIMSAS2 Mode: SATA
Firmware Settings
The following settings are what I chose to set based on a very quick session search Duckduckgo for explanations. The built-in documentation is somewhat lacking although there are URLs (encoded as QR codes) for more details. This is one area where firmware authors should pay more attention – even if they just hinted which settings work best for Windows, which work best for Linux, and which ones are for compatibility for older hardware.
The choices I’ve made may not be the best, but it seems to be working. Some of the explanations may be off, so I’d welcome corrections. All of these settings are found under the “Advanced” tab of the firmware page :-
CPU Configuration
SMT: Or “hyperthreading”. It is possible some scientific computing workloads might work better with this turned off, but my recommendation is to leave it to “Auto”.
CPB – Core performance boost: presumably allows one core to accelerate when other cores are idle. Left on “Auto”.
Global C-State control: related to power-saving. There’s a suggestion that disabling this may result in extra stability. Disabled.
Local APIC Mode: controls how the APIC appears to the operating system with choices of Auto, Compatible, xAPIC, or 2xAPIC. Supposedly 2xAPIC allows for greater efficiency on higher core counts. Set to 2xAPIC.
L1 Stream HW Prefetcher: Enables or disabled pre-fetching memory into cache. Enabled.
L2 Stream HW Prefetcher: Enables or disabled pre-fetching memory into cache. Enabled.
SMEE (SME?): Secure memory (i.e. encrypted) for virtual machines. Not likely to make much difference in my case as I’m the exclusive owner of both the “host” and all of the virtual machines running on it. Left as “Auto”.
SEV-ES ASID Space Limit Control: More on virtual machine security. Left on Auto.
SVM mode: This option seemed to disappear on the upgrade to 7.09. If this does appear, enable it.
ROM Armor: protection for SPI flash. Left as Enabled.
Chipset
IOMMU: virtual machine I/O virtualisation to allow PCIe pass-through to a virtual machine. Enabled.
ACS: More I/O virtualisation. Suggestions hinting at allowing PCIe←→PCIe transfers. Some hints at better IOMMU set up. Enabled.
Enable AER Cap: PCIe error handling. Presumably disabling Linux AER error handling. Disabled.
PCIe ARI Support: Enables support for ARI which allows a device to more easily support pretending to be multiple devices (so a graphics card could be shared amongst multiple virtual machines). Although card support for this is probably quite rare, I enabled it anyway.
PCIe Ten Bit Tag Support: Allows a supporting device to use greater bandwidth and lower latency. Enabled.
NUMA node(s) per socket: It is suggested that this allows the processor’s CCXes (the ‘core complex’ that appears as individual chiplets in an AMD processor) to operate as separate NUMA nodes. Set to NPS4.
ACPI SRAC L3 Cache as NUMA domain: It is suggested that this also allows each CCX to function as a NUMA node. Enabled.
TSME: Or Transparent SME. Support for SME is done by the firmware rather than the OS. Disabled.
HPET: High Precision Timer. Enables support for a newer way of doing timing. Enabled.
… (missing details because they weren’t of interest to me)
SLIMSAS1 Mode/SLIMSAS2 Mode: As mentioned previously, allows switching the SlimSAS ports from supporting NVME devices to supporting SATA devices. Switched to SATA mode!
PCI
PCI latency timer: How many clock cycles a 32-bit PCIe card can hang onto the bus for. Leave alone (32 cycles).
PCI-X latency timer: How many clock cycles a 64-bit PCIe card can hang onto the bus for. Leave alone.
VGA Palette Snoop: Whether to allow other cards to snoop on the VGA palette which is used by older cards for video encoding and the like. Disabled.
PERR# Generation: Something to do with PCIe card errors. Left alone.
SERR# Generation: Something to do with PCIe card errors. Left alone.
Above 4G Decoding: Allows card to specify a 64-bit address to house their memory window. Enabled.
Re-size BAR Support: Allows a card to negotiate a larger address window than the default of 256Mbytes. Enabled.
SR-IOV Support: Where PCIe cards allow, enables the creation of virtual devices to be allocated to virtual machines. Enabled.
BME DMA Mitigation: Re-enable Bus Master Attribute after SMM is locked. Whatever that means! Left disabled.
So two days ago, I upgraded my main workstation to Ubuntu 23.10; a few little issues (mostly related to my own scripts), but nothing serious. Yet.
On the following day, my smart TV box started misbehaving. It couldn’t see any of the videos NFS mounted from my workstation, ITVX threw up a website error (this should have been a clue), but Youtube worked fine (which showed that the network was working fine).
So I did the obvious thing and started checking the NFS parameters to see if anything had changed. Nothing definite but on the way I noticed that the TV box wasn’t getting an IPv4 address from the dhcp server; IPv6 was working fine but some services don’t work on an IPv6 network.
I foolishly assumed that the TV box had stopped requesting addresses via dhcp – backed by the dhcp logs which showed no requests had been logged since the previous day. Set a static address, and everything sprang into life (except for ITVX who seem to have decided that only approved TV boxes should be allowed to run their code).
Later that same day, I upgraded a switch which failed to come back (“Failed to adopt”) which caused a daisy-chained wireless access point to disappear (“Failed to adopt”). And then a little while later, a second unconnected wireless access point also disappeared.
After a few reboots of the switch (and access points), I finally checked the dhcp server and found that its root filesystem had become ‘read-only’. But that wasn’t the end of the misdiagnosis …
I assumed that the SD card in my dhcp server (a tiny ARM box) was fried, so made arrangements to backup the contents, buy a couple of replacements, and try a spare (which was broken). After the spare turned out to be broken, I ran fsck on the root filesystem of the original and a whole bunch of errors were fixed.
Re-installed into the ARM box, and everything sprang to life again.
I guess the moral of the story is that you should check the basic services before diving into making assumptions.
I use technologies like cookies to store and/or access device information. I do this to improve browsing experience and to show (non-) personalised ads. Consenting to these technologies will allow me to process data such as browsing behaviour or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.