More Zonkyness

This is a collection of notes from my upgrade to an ASRock TRX50 WS motherboard fitted with an AMD Threadripper 7970X processor (32 cores) and 256Gbytes of memory. The upgrade meant that I retained the case, drives, graphics card, etc. from the previous system.

Most of the problems encountered were due to user stupidity.

First of all, whilst many of us have heard about the amount of time that DDR5 takes to “calibrate” itself, what I didn’t know was that the firmware status code shows “00” during this process (a dedicated “I’m messing with memory” code would be handy). And whilst it takes a while to do, if it takes longer than about 5m, then something else is wrong.

In my case it turned out that I hadn’t read the instructions properly and I hadn’t connected enough power connectors. To get it to work, I needed the usual 24-pin power connector, an 8-pin connector, and a 6-pin connector all connected on the “drive” side of the motherboard (opposite the side with the PCIe slots). Once that was sorted, the system was up and running.

The remaining notes relate to “tweaking”.

Booting Linux

Of course I use Linux, what the hell else would I use? FreeBSD? Well, that would be a good choice.

The biggest problem I had booting Linux was changing the netplan configuration to pick up the new network interfaces. In my case, the Marvell interface (the 10G one) came up as enp65s0 and the Realtek interface (the 2.5G one) as enp69s0. Because I’m bound to plug the cable into the wrong interface, I simply bonded the two interfaces together; the relevant section of my netplan configuration is as follows :-

network:
  version: 2
  renderer: networkd
  bonds:
    james:
      interfaces: [enp65s0, enp69s0]
  ethernets:
    enp65s0: {}
    enp69s0: {}
  bridges:

Yes, you can choose silly names here. And yes the bonding works fine – just now I swapped the cable over to the “right” NIC with numerous active network connections, and everything stayed alive.

Firmware Upgrade

The motherboard was supplied with version 6.04 of the firmware (I refuse to call this a “BIOS” because it just isn’t “basic” any more) whereas the latest was 7.09. The process is fairly simple :-

1. Download the relevant firmware version from https://www.asrock.com/mb/AMD/TRX50%20WS/index.asp#BIOS.
2. Save it to a FAT32 USB disk – I used a vfat formatted disk and I have a sneaking suspicion that exFAT will work too. The “Instant Flash” instructions by ASRock are obviously somewhat dated – it even mentions that saving to a floppy disk will work!
3. Reboot the system and start the UEFI firmware. Select “Tools” and “Instant Flash”.
4. Follow the on-screen instructions.

If you’re replacing a motherboard you won’t need detailed instructions here, but it is worth mentioning that the process takes a couple of reboots, and the second involves doing that memory calibration thing, so it takes an unusually long time to start.

I didn’t go to the effort to time the whole process, but my system went down at 18:04 and was back up at 18:15. So roughly 10 minutes.

SlimSAS

This isn’t currently 100% confident as I haven’t plugged anything in yet (ignoring a failed attempt when I assumed it work just work), but the SlimSAS ports can be configured for SATA mode in the firmware. Just go to Advanced, Chipset, go to the end of the list (which involves scrolling) past the settings for the PCIe slot configuration parameters and set :-

1. SLIMSAS1 Mode: SATA
2. SLIMSAS2 Mode: SATA

Firmware Settings

The following settings are what I chose to set based on a very quick session search Duckduckgo for explanations. The built-in documentation is somewhat lacking although there are URLs (encoded as QR codes) for more details. This is one area where firmware authors should pay more attention – even if they just hinted which settings work best for Windows, which work best for Linux, and which ones are for compatibility for older hardware.

The choices I’ve made may not be the best, but it seems to be working. Some of the explanations may be off, so I’d welcome corrections. All of these settings are found under the “Advanced” tab of the firmware page :-

CPU Configuration

1. SMT: Or “hyperthreading”. It is possible some scientific computing workloads might work better with this turned off, but my recommendation is to leave it to “Auto”.
2. CPB – Core performance boost: presumably allows one core to accelerate when other cores are idle. Left on “Auto”.
3. Global C-State control: related to power-saving. There’s a suggestion that disabling this may result in extra stability. Disabled.
4. Local APIC Mode: controls how the APIC appears to the operating system with choices of Auto, Compatible, xAPIC, or 2xAPIC. Supposedly 2xAPIC allows for greater efficiency on higher core counts. Set to 2xAPIC.
5. L1 Stream HW Prefetcher: Enables or disabled pre-fetching memory into cache. Enabled.
6. L2 Stream HW Prefetcher: Enables or disabled pre-fetching memory into cache. Enabled.
7. SMEE (SME?): Secure memory (i.e. encrypted) for virtual machines. Not likely to make much difference in my case as I’m the exclusive owner of both the “host” and all of the virtual machines running on it. Left as “Auto”.
8. SEV-ES ASID Space Limit Control: More on virtual machine security. Left on Auto.
9. SVM mode: This option seemed to disappear on the upgrade to 7.09. If this does appear, enable it.
10. ROM Armor: protection for SPI flash. Left as Enabled.

Chipset

1. IOMMU: virtual machine I/O virtualisation to allow PCIe pass-through to a virtual machine. Enabled.
2. ACS: More I/O virtualisation. Suggestions hinting at allowing PCIe←→PCIe transfers. Some hints at better IOMMU set up. Enabled.
3. Enable AER Cap: PCIe error handling. Presumably disabling Linux AER error handling. Disabled.
4. PCIe ARI Support: Enables support for ARI which allows a device to more easily support pretending to be multiple devices (so a graphics card could be shared amongst multiple virtual machines). Although card support for this is probably quite rare, I enabled it anyway.
5. PCIe Ten Bit Tag Support: Allows a supporting device to use greater bandwidth and lower latency. Enabled.
6. NUMA node(s) per socket: It is suggested that this allows the processor’s CCXes (the ‘core complex’ that appears as individual chiplets in an AMD processor) to operate as separate NUMA nodes. Set to NPS4.
7. ACPI SRAC L3 Cache as NUMA domain: It is suggested that this also allows each CCX to function as a NUMA node. Enabled.
8. TSME: Or Transparent SME. Support for SME is done by the firmware rather than the OS. Disabled.
9. HPET: High Precision Timer. Enables support for a newer way of doing timing. Enabled.
10. … (missing details because they weren’t of interest to me)
11. SLIMSAS1 Mode/SLIMSAS2 Mode: As mentioned previously, allows switching the SlimSAS ports from supporting NVME devices to supporting SATA devices. Switched to SATA mode!

PCI

1. PCI latency timer: How many clock cycles a 32-bit PCIe card can hang onto the bus for. Leave alone (32 cycles).
2. PCI-X latency timer: How many clock cycles a 64-bit PCIe card can hang onto the bus for. Leave alone.
3. VGA Palette Snoop: Whether to allow other cards to snoop on the VGA palette which is used by older cards for video encoding and the like. Disabled.
4. PERR# Generation: Something to do with PCIe card errors. Left alone.
5. SERR# Generation: Something to do with PCIe card errors. Left alone.
6. Above 4G Decoding: Allows card to specify a 64-bit address to house their memory window. Enabled.
7. Re-size BAR Support: Allows a card to negotiate a larger address window than the default of 256Mbytes. Enabled.
8. SR-IOV Support: Where PCIe cards allow, enables the creation of virtual devices to be allocated to virtual machines. Enabled.
9. BME DMA Mitigation: Re-enable Bus Master Attribute after SMM is locked. Whatever that means! Left disabled.