A Linux Evening

There are 3 type of person who knows a lot about cars:

1. A Car mechanic 2. People who love cars, and tinker them constantly 3. People who have a shitty car and something always breaks.

Personally I'm not a car guy, I treat them as tools. I can do this because I always had a reliable car. I have a friend who once mocked me for my inexperience. I had to reminding him why he knew so much about cars. He was a Type 3 guy. No a shitty car tho, but something always broke on it anyway.

So getting back to you, can we just have something Works? Is that a high bar?

I'm so glad that my hit-and-run post has been so useful. After seeing Fabien's blog post I did a quick search and it turns out that the solution has spread fairly broadly to other forums. My choice of 0x33 was arbitrary so makes a nice canary for seeing it spread out. I'm thrilled. Sharing experiences and solutions is so essential to learning. I've benefited enormously from the generosity of open source developers and communities and from individuals documenting pieces of their projects, glad to have raised the ocean a little in return.

My use case was (and remains) having a Xilinx Artix 7 FPGA in an external Thunderbolt 3 enclosure for testing the development of DSP accelerators using open source tooling. I didn't want to have the FPGA board inside the PC to be able to swap it to my laptop easily, because it produces a lot of heat, and so when I misused the PCIe soft core (litePCIe: https://github.com/enjoy-digital/litepcie/) it doesn't take down the OS. Being able to reload the FPGA and effectively hotplug the device has been very helpful.

Since I knew my issue was around hotplugging I searched for information around PCIe hotplugging and I think (it was two years ago...) that I found the answer from one of these two threads. Both mention the option of reserving PCIe addresses for hotplug busses as a workaround, and a workaround was all I needed.

https://www.spinics.net/lists/linux-pci/msg64841.html

https://review.coreboot.org/c/coreboot/+/35946

dmesg and the various kernel logs are my first stop for any odd behavior on Linux. Especially with any state change to a device (plugging in, turning on, removing, reconfiguring etc) the kernel logs tend to give invaluable info.

I had already been looking at eGPU forums to choose the Thunderbolt 3 enclosure (ended up with the ORI-SCM2T3-G40-GY) and there were various discussions of hotplugging issues there, but I don't think I found the specific kernel options to fix it there.

Check out this docs page for the kernel parameters: https://docs.kernel.org/admin-guide/kernel-parameters.html

For the string "pci=realloc,assign-busses,hpbussize=0x33"

`realloc` Enable/disable reallocating PCI bridge resources if allocations done by BIOS are too small to accommodate resources required by all child devices.

assign-busses [X86] Always assign all PCI bus numbers ourselves, overriding whatever the firmware may have done.

`hpbussize=nn` The minimum amount of additional bus numbers reserved for buses below a hotplug bridge. Default is 1.

0x33 (decimal 51) is arbitrary, but large enough that I was unlikely to ever exhaust that address range even with a large number of devices chained together on the Thunderbolt bus. I think (though would have to check) that the address space does get exhausted with multiple hotplug cycles. I haven't hit that issue since I shutdown the computer close to daily to save power.

Sadly, there's no Segway, those things are expensive and I have a long wish list before reaching that point. Currently I'm saving up to get a microscope, I have a large box of RF integrated circuits that I'd like to do some show and tell with on Twitch. :) Also no unmarked 40% keyboard, RSI means I'm a total fanboy of the Microsoft Ergonomic keyboard and the Anker vertical mouse. Cheap and so comfortable. I have done some kernel compiling, mostly to learn more about kernel modules as I've been trying to make the learning curve and user experience of experimenting with FPGAs over PCIe easier. If anyone has some experience with DKMS and creating Debs I'd welcome a chance to chat. Ditto if there's any Debian maintainer with experience packaging kernel modules, I made some headway a while back to repackage linux-gpib but got stalled out on a few of the details of maintaining patches against the upstream.

Cheers and Happy Holidays!