A while back at a SLUG (Siouxland Linux Users Group) meeting, an attendee had some Lenovo ThinkCentre small form factor computers to sell at very reasonable prices. They were surplus that I believe a school used as thin clients. It turned out that one of them would not boot and I thought that sounded like an interesting challenge to try to undertake, and a good excuse to flex a technical muscle that I hadn't gotten to use in a while.

The model in question was the Lenovo ThinkCentre M720q (user guide and hardware manual, family spec sheet) which I luckily had two very similarly configured units with i5-8400T processors that also happened to be one serial number apart! It was helpful to have so similar a working unit to compare and test behaviors against from the broken one. For example, I was able to remove the RAM from the working unit and get a beep pattern that corresponded to an error in the manual about checking the RAM. This could serve as a fairly early signal from the broken system to determine just how far the start up process was getting without having some actual instrumentation like a serial console.

I had considered getting the RS232 expansion module (Lenovo Part Number 04X2733, available in their parts site) but decided, after being directed to some schematics online, that getting useful diagnostic information from the interface might not be practical. There seemed to be options to get both the CPU and PCH debug serial ports over the "COM1" serial port, but for one path it seemed to require populating some 0402 strap resistors, and another path even more components. I opted to do a little more software digging instead.

After reviewing many discussion threads, including:

• https://www.badcaps.net/forum/troubleshooting-hardware-devices-and-electronics-theory/troubleshooting-desktop-motherboards-graphics-cards-and-pc-peripherals/bios-schematic-requests/3285056-lenovo-m920x-eq370-nm-b551-iq3x0il-type-10s2-bios
• https://forums.servethehome.com/index.php?threads/lenovo-thinkcentre-thinkstation-tiny-project-tinyminimicro-reference-thread.34925/
• https://www.reddit.com/r/techsupport/comments/x5jryj/lenovo_m710q_tiny_stuck_at_post_after_bios_update/
• https://www.reddit.com/r/Lenovo/comments/18bupxo/how_the_hell_do_you_update_bios_on_m720q_with_no/

I learned that there are multiple ThinkCentre models that use the NM-B551 PCB, including the M720q. This led to more insight about how to approach recovering a unit that doesn't boot. As well as that some AMI Aptio V distributions include a lower level recovery stub boot loader that can read and flash a BIOS image from a USB drive when a key combination is held down during boot. I'm not sure how far the system has to start up, but in looking for a citation for the filename and key combination needed, as well as trying to discern the filename from the most recent BIOS update file, I was left just trying the default AMIBOOT.ROM and Ctrl-Home without any luck. I also wasn't sure my failed unit was executing enough of the UEFI software to execute this critical recovery stub. But it was an avenue that seemed worth pursuing at least that far.

From the previous batch of reading I'd also been introduced to the idea of transplanting a working SPI flash image onto a non-working board in order to restore it to a working state. The strongest reason I can think of that might require this seems to be a partially written BIOS update. Power loss during the BIOS update process seemed the most likely explanation for why this unit no longer seemed to do anything, even when booted with no RAM installed. This stage of the process also involved testing the main system components, namely the CPU and RAM, in the working unit. Everything worked fine when moved over to the known good test unit, which directed the investigation back toward the board or BIOS. And so a Pamona 5250 SOIC-8 test clip was ordered.

Repurposing an old Raspberry Pi from a past project turned out to be pretty simple. Migrating the sources.list from raspbian.raspberrypi.org to archive.raspberrypi.org was the biggest hurdle to installing flashrom to actually perform the read.

pi@raspberrypi:~ $ flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=512
flashrom v0.9.9-r1954 on Linux 4.14.79-v7+ (armv7l)
flashrom is free software, get the source code at https://flashrom.org

Calibrating delay loop... OK.
Found Winbond flash chip "W25Q128.V" (16384 kB, SPI) on linux_spi.
No operations were specified.

After backing up the broken unit's SPI flash, and reading the working unit's, it was simple enough to write the working image onto the broken unit's SPI flash chip.

pi@raspberrypi:~ $ time flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=512 -w m720q-working01.bin
flashrom v0.9.9-r1954 on Linux 4.14.79-v7+ (armv7l)
flashrom is free software, get the source code at https://flashrom.org

Calibrating delay loop... OK.
Found Winbond flash chip "W25Q128.V" (16384 kB, SPI) on linux_spi.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

real    17m49.956s
user    0m35.689s
sys     0m13.801s

Replacing the CPU and RAM that had been tested resulted in the computer booting right up! I expected that all of the unit specific details (UUID, serial number, MAC address, etc.) would be copied since most of those details live in the SPI flash. This turned out to be the case and that became the next task: applying the identifying information from the broken unit to a copy of the working unit's SPI flash dump.

The most obvious detail was the unit serial number. Finding it as ASCII data in the SPI flash dump was pretty simple, as was changing the last byte. Next up was the onboard Intel I219-V Ethernet MAC address. I could get the working unit's MAC address easily enough and find those bytes in the SPI flash dump, then correlate those locations to the broken unit's dump to see what value it was supposed to have. The first occurrence was suspiciously at offset 0x1000 in the dump. I stumbled into instances that had the vendor and device portions separated by 0xFF 0xFE, which seemed strange. I also managed to find that some instances had the locally administered bit set, so instead of a Vendor ID of 8C-16-45 they were 8E-16-45. Some of these were data bytes and some occurrences were in ASCII encoded hex. One additional fun detail I happened upon was that despite my unit serial numbers being consecutive, the MAC address values were not! The working, lower numbered, unit's device ID portion was 99-63-22 while the broken, higher numbered, unit's was 99-62-F6. This seems to suggest either that the board MAC address is provisioned before the unit serial number, or maybe that MAC address barcodes are printed multiple to a sheet and perhaps the values were consumed in columns instead of row by row. Puzzling about this kind of manufacturing process detail was pretty fun.

Recalling that one of the forum posts discussed NVAR entries, and encountering several while working through the serial number and MAC address changes, I thought to be more thorough in my efforts to find unit specific details I had been unaware of trying to uncover. The pretty coarse strategy of using strings -a and vimdiff to compare the SPI flash dumps proved quite fruitful. After first dumping the strings to an intermediate file, and having built a list of "variable-like" identifiers to spot-check (NVAR, DmiVar, BIOS_PARAMETER, BONVAR) I repeatedly did that with the following:

vimdiff <(grep -C 5 NVAR m720q-working-strings.txt) <(grep -C 5 NVAR m720q-broken-strings.txt)

This process found more MAC address values as well as the embedded Windows product key, some variables I was worried about but decided to ignore (UnlockID, UnlockIDCopy, OfflineUniqueIDEKPub, OfflineUniqueIDEKPubCRC, VsmLocalKey2, VsmLocalKeyProtector) and BONVAR provided a narrower review of the NVAR values, since I'd missed that NVAR also matches BONVAR… After deciding I was sufficiently satisfied with my work I decided to test out this modified working unit SPI flash dump on the broken unit.

pi@raspberrypi:~ $ time flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=512 -w m720q-working-mod.bin
flashrom v0.9.9-r1954 on Linux 4.14.79-v7+ (armv7l)
flashrom is free software, get the source code at https://flashrom.org

Calibrating delay loop... OK.
Found Winbond flash chip "W25Q128.V" (16384 kB, SPI) on linux_spi.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

real    14m35.587s
user    0m3.804s
sys     0m9.657s

This also booted up! The MAC address, serial number, and other system specific values seemed to have been successfully updated in the BIOS UI. After booting into Finnix to see if the OS view of the MAC address was consistent I ran into a problem. The onboard NIC was not showing up in user space. Then I saw this in dmesg:

[    2.249820] e1000e 0000:00:1f.6: The NVM Checksum Is Not Valid
[    2.299688] e1000e: probe of 0000:00:1f.6 failed with error -5

I was able to fix the MAC address after finding this Super User question which pointed me to the Intel BootUtil tools which were able to satisfy the thing that I had failed to:

root@0:~/intel/APPS/BootUtil/Linux_x64# ./bootutil64e -NIC 1 -defcfg
Error: Connection to QV driver failed - please reinstall it!

Intel(R) Ethernet Flash Firmware Utility
BootUtil version 1.43.32.0
Copyright (C) 2003-2026 Intel Corporation

Setting PXE EEPROM words back to defaults on NIC 1...done

Port Network Address    Location Series  WOL Flash Firmware             Version
==== ============ ============== ======= === ========================== =======
   1 8C16459962F6 00000:000:31.6 Gigabit N/A FLASH Not Present

Despite being confused by "FLASH Not Present" it seems that things were now happier:

root@0:~/intel/APPS/BootUtil/Linux_x64# rmmod e1000e
root@0:~/intel/APPS/BootUtil/Linux_x64# modprobe e1000e
root@0:~/intel/APPS/BootUtil/Linux_x64# dmesg | tail
[ 1131.985504] IPv6: ADDRCONF(NETDEV_CHANGE): enx3c18a04043d8: link becomes ready
[ 1131.985764] r8152 2-1:1.0 enx3c18a04043d8: carrier on
[ 1720.749439] e1000e: Intel(R) PRO/1000 Network Driver
[ 1720.749441] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[ 1720.749540] e1000e 0000:00:1f.6: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[ 1720.961378] e1000e 0000:00:1f.6 0000:00:1f.6 (uninitialized): registered PHC clock
[ 1721.032170] e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) 8c:16:45:99:62:f6
[ 1721.032175] e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
[ 1721.032296] e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
[ 1721.042869] e1000e 0000:00:1f.6 eno1: renamed from eth0
root@0:~/intel/APPS/BootUtil/Linux_x64# ifconfig -a
eno1: flags=4098<BROADCAST,MULTICAST>  mtu 1500
        ether 8c:16:45:99:62:f6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xb1200000-b1220000
...

But I would find that I had to reboot the system before packets would flow across the interface. Then things were pretty happy! I ran it for a few hours and monitored sensors output and noticed fresh thermal grease showed 4-5°C lower temperatures for the CPU cores as compared to the always working unit.

After a little more testing I finally noticed the System UUID was the same on both units and the ME (Management Engine) FW Version was 0.0.0.0 on the previously broken unit. This didn't seem like a big problem but it would be preferable to have it working and happy. Tracking down the UUID was a bit of a challenge because the first 4 chunks were in little endian while the last 6 byte chunk was in big endian. At least that made the data easier to find in the dump, and I believe it was in 3 distinct locations.

The Management Engine firmware was something of a saga on its own. I had seen, but not read carefully, threads on the Win-Raid forum about cleaning the ME Firmware when cloning a BIOS dump from one machine to another. I was even using MEAnalyzer to compare the working and broken SPI flash dumps quite early in the process. But it didn't show meaningful differences besides the broken unit's version being slightly newer. That led me to imagine that the problem may have been a power loss during a BIOS update, resulting in an incomplete update that left the system not bootable.

This thread proved incredibly helpful and provided a framework for digging into the problem of analyzing differences between the two SPI flash dumps. The report generated by UEFITool was also pretty informative by exploring the UEFI image format and showing the GbE Region which I would come to find was what the Intel BootUtil had regenerated. The reason it had been unhappy was because I updated the MAC address, which was the first 6 bytes of the block starting at 0x1000, but not the checksum at the end of the block. That and the ME Firmware Version problem were fixed quite completely when I followed the procedure outlined in the cleaning guide, which was essentially:

1. use the Intel Flash Image Tool to extract the image regions
2. replace the ME Firmware region in the first extraction with the clean one from the archive that matches the extracted one being replaced
3. use the Intel Flash Image Tool to generate a new UEFI SPI flash image

I took the working unit's modified SPI flash image and extracted the regions, and then extracted the broken unit's SPI flash image regions. I then opened the working image again in the Intel Flash Image Tool, swapped in the ME Firmware region from the archive, the GbE Region from the broken image, and the BIOS Region from the modified image that had the System UUID also fixed. This was then flashed onto the broken unit's SPI flash:

pi@raspberrypi:~ $ time flashrom -p linux_spi:dev=/dev/spidev0.0,spispeed=512 -w outimage.bin
flashrom v0.9.9-r1954 on Linux 4.14.79-v7+ (armv7l)
flashrom is free software, get the source code at https://flashrom.org

Calibrating delay loop... OK.
Found Winbond flash chip "W25Q128.V" (16384 kB, SPI) on linux_spi.
Reading old flash chip contents... done.
Erasing and writing flash chip... Erase/write done.
Verifying flash... VERIFIED.

real    16m56.755s
user    0m30.201s
sys     0m9.965s

It started up, all the identifying information seemed correct, the ME Firmware Version was no longer 0.0.0.0, and the onboard Intel I219-V network interface worked without the BootUtil shenanigans. This seemed like the way.

I built a new PC a few years ago using the Intel 8086K that I won in the Intel 40th Anniversary Sweepstakes. It's been a good machine that I've only really used for games. Over the last year or so I've used it for some more hobby projects and I finally got annoyed with the Downloads special folder always opening up with the contents being grouped by date.

I wasn't terribly surprised to find that this was a common thing people complained about. In the Microsoft Community thread I found about the problem the solution was to basically forcibly set all folder view settings. This is a reasonably straight forward answer and what I believe I'd done on my previous Windows installs. After all, details view is the only right answer, right? :) In the Super User question on the topic it seemed that there was a nicer solution that preserved the existing folder view settings. It also required taking ownership of the key from regedit.exe. But then:

> reg query 'HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderTypes\{885a186e-a440-4ada-812b-db871b942259}\TopViews\{00000000-0000-0000-0000-000000000000}' /v GroupBy

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderTypes\{885a186e-a440-4ada-812b-db871b942259}\TopViews\{00000000-0000-0000-0000-000000000000}
    GroupBy    REG_SZ    System.DateModified

> reg add 'HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderTypes\{885a186e-a440-4ada-812b-db871b942259}\TopViews\{00000000-0000-0000-0000-000000000000}' /v GroupBy /t REG_SZ /d System.Null /f
The operation completed successfully.

> reg query 'HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderTypes\{885a186e-a440-4ada-812b-db871b942259}\TopViews\{00000000-0000-0000-0000-000000000000}' /v GroupBy

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FolderTypes\{885a186e-a440-4ada-812b-db871b942259}\TopViews\{00000000-0000-0000-0000-000000000000}
    GroupBy    REG_SZ    System.Null

However, even after restarting explorer.exe my Downloads folder was still showing up grouped.

The solution came from another forum thread on the topic which explained the way view settings for special folders are stored in the registry and presented a PowerShell snippet to remove them for the Downloads special folder:

$Bags = 'HKCU:\Software\Classes\Local Settings\Software\Microsoft\Windows\Shell\Bags'
$DLID = '{885A186E-A440-4ADA-812B-DB871B942259}'
(Get-ChildItem $bags -recurse | ? PSChildName -eq $DLID ) | Remove-Item
gps explorer | spps

This also restarted explorer.exe which may not be a surprise, but I don't use PowerShell enough to know gps is Get-Process and spps is Stop-Process. But finally my Downloads folder was showing up in details view sorted by modified like I wanted!

Recently there has been news of Google deploying advertisements more invasive than content banners in their Google TV launcher. Upon experiencing this directly it seemed like a good idea to dig into replacing the launcher.

The first step was testing out alternative launchers. I tried Projectivy Launcher and FLauncher and ended up preferring FLauncher, it is very simple and it's open source which is an added nicety.

The next step was figuring out how to set the default launcher. The most conservative approach was to do some button remapping which led me to Button Mapper which didn't actually work on my Chromecast since it, at least with the free version, didn't even list the Home button as a remappable key. I could add it as a custom button but testing was going to require the paid upgrade. That, coupled with the extra step at every start up of going from the default launcher to the preferred launcher, made more investigation seem like the right answer.

Eventually I came across a guide to disabling the Google TV Launcher which also covered choosing a new default! It basically boils down to using adb commands to disable 2 apps:

pm disable-user --user 0 com.google.android.apps.tv.launcherx
pm disable-user --user 0 com.google.android.tungsten.setupwraith

and then choosing a new default once you press the Home button.

I've only ever used adb with a USB connection. Another guide I came across made mention of ADB TV which is an app that allows running adb commands from the Android environment running on the Google TV. This made it easy to disable the two apps listed.

However, I only disabled the launcherx app. I then exited ADB TV and landed at a black screen. This was rather nerve-racking and after rebooting and pushing various buttons the only response I could get was from the dedicated YouTube and Netflix buttons launching their respective apps. However, pressing back a few times from that put me back in FLauncher for a reason I can't explain. Then, I could re-open ADB TV and disable the setupwraith app which allowed me to choose a new default launcher by pressing the Home button. All told following the guide would have been much easier. Oops.

Everything is working nicely now though!

Additional links:

• https://www.reddit.com/r/AndroidTV/comments/1aefuo6/new_ad_hell_carls_jr_homescreen_ad_with_auto_play/
• https://www.reddit.com/r/Chromecast/comments/1ae37sa/full_screen_video_ads_wtf/
• https://www.reddit.com/r/AndroidTV/comments/154dqd1/ive_had_enough_of_google_chromecast_w_google_tv/

This week at work I was faced with a bug that manifested as Windows XP clients being slower to access parts of a web interface than Windows 7 or 10. This was strange because in Wireshark the requests looked basically the same.

It turned out that the embedded system hosting the web interface was rejecting Ethernet frames larger than 1500 bytes. This was most likely because of a misinterpretation of the MTU as referring to the frame size (at layer 2) instead of the payload size (at layer 3).

The strangest part about this issue was just how consistent the effect was across all browsers tested given a version of Windows. Windows XP took nearly 22 seconds to complete a request that took just over 2 seconds in Windows 7. The core of this behavior ended up being a side-effect of the TCP retransmission timeout.

The default retransmission timeout for Windows XP is stored in the TCPInitialRTT registry value for a given network adapter. The default value when the registry value does not exist is 3000ms. This aligned with the observation under XP that the first retransmission occurred after 3 seconds with subsequent retransmissions occurring at 6 and 12 seconds. The first retransmission was sent unfragmented while the second and third retransmissions were a fragmented version of the original frame, which was ultimately accepted. The fragmented payloads were at most 576 bytes which seemed like an interesting size but I did not investigate.

Windows 7 on the other hand appears to retry after 300 milliseconds, with subsequent retries at 600ms and 1200ms. But the same basic behavior was followed: the first retry was the full payload while the subsequent retries were fragmented into at most 576 byte payloads. The key difference appearing to be the retransmission timeout. This masked the underlying issue for Windows 7 clients.

While I did not search extensively, I did not find an explanation for either the doubling back-off of the retransmissions or the 576 byte payload size.

The rather quick fix was to change the Linux driver to accept frames up to the actual size that could be handled. And introducing an error message if a frame is rejected for length reasons.

A friend asked several people on irc about NameSilo's API and dynamic DNS entries. He found a PowerShell script to update a subdomain with the current IP address of the system running the script. The subdomain detail was the crux of the question: how to get it to update a "naked" domain. Several of us read through the API reference but the dnsUpdateRecord function didn't explain how to update the base domain's A record.

It turned out that simply leaving off the rrhost parameter was sufficient to get the job done.

After we were done iterating on it, we had a PowerShell function to update any record. Including enough intelligence to handle the base domain case. I don't believe non-A records were tested, but it met the need of updating the base domain and a sub-domain or two from a scheduled task.

# NameSilo API Dynamic DNS
# Variables
$APIkey = ""
$domain = ""

function NameSilo-dnsUpdateRecord {
	param ([string]$APIKey, [string]$Domain, [string]$Record, [string]$Type)

	# Retrieve the DNS entries in the domain.
	$listdomains = Invoke-RestMethod -Uri "https://www.namesilo.com/api/dnsListRecords?version=1&type=xml&key=$APIkey&domain=$domain"
	$Records = $listdomains.namesilo.reply.resource_record | where { $_.type -eq $Type }

	$UpdateRecord = $null
	$IsNaked = $False
	foreach ($r in $Records ) {
		if ([string]::IsNullOrEmpty($Record) -and $r.host -eq $Domain) {
			$UpdateRecord = $r
			$IsNaked = $True
			break
		} elseif ($r.host -eq "$($Record).$($Domain)") {
			$UpdateRecord = $r
			break
		}
	}
	if ($UpdateRecord -eq $null) {
		echo "Error: Could not find requested record: $($Record).$($Domain)"
		Exit
	}

	$CurrentIP = $listdomains.namesilo.request.ip
	$RecordIP = $UpdateRecord.value
	$RecordID = $UpdateRecord.record_id

	# Only update the record if necessary.
	if ($CurrentIP -ne $RecordIP){
		$url = "https://www.namesilo.com/api/dnsUpdateRecord?version=1&type=xml&key=$APIkey&domain=$Domain&rrid=$RecordID"
		if ($IsNaked -eq $False) {
			$url += "&rrhost=$record"
		}
		$url += "&rrvalue=$CurrentIP&rrttl=3600"
		$update = Invoke-RestMethod -Uri $url
	} else {
		echo "IP Address has not changed."
	}
}

# Invocations:
NameSilo-dnsUpdateRecord -APIKey $APIkey -Domain $domain -Record "" -Type "A"
NameSilo-dnsUpdateRecord -APIKey $APIkey -Domain $domain -Record "*" -Type "A"
NameSilo-dnsUpdateRecord -APIKey $APIkey -Domain $domain -Record "test" -Type "A"

Since I don't do hardly anything in PowerShell, aside from trying to use it more than cmd.exe on Windows because it is a resizable window, I did a little more reading after this was written and concluded that it is not likely representative of PowerShell best practices.

But it is posted here just in case it might be useful to someone.