How Facebook deals with PCIe faults to keep its data centers running reliably
1–10 of 53 posts
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#2Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc..
EDIT: There is a libpci for python that uses pciutls...
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#3Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc.. EDIT: There is a libpci for python that uses pciutls...
[deleted]
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#4Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc.. EDIT: There is a libpci for python that uses pciutls...
Aren’t pciutils dependent on hardware support though? I have a couple of Asus mobos that have weird mouse movement depending on how slightly angled the GPU is in the slot - I tried to use pciutils to diagnose the issue but the “gamer”-grade Core i7 I have in there wasn’t on support-list because Intel only enables PCIe diags on i9/X and Xeon chips.
...and I know a few DCs that loaded up on cheap low-grade i3/i5 chips for “near-line” computing (slow but cheap and cheerful) so they’d be SoL too.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#5Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#6Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.
[deleted]
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#7Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc.. EDIT: There is a libpci for python that uses pciutls...
Aren’t pciutils dependent on hardware support though? I have a couple of Asus mobos that have weird mouse movement depending on how slightly angled the GPU is in the slot - I tried to use pciutils to diagnose the issue but the “gamer”-grade Core i7 I have in there wasn’t on support-list because Intel only enables PCIe diags on i9/X and Xeon chips. ...and I know a few DCs that loaded up on cheap low-grade i3/i5 chips…
...Why would sysfs expose something that your hardware doesn't support while pciutils doesn't?
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#8> Bad…link speed…and bad…link width…were other concerning PCIe faults. These faults can be difficult to detect without some sort of automated tool…
Most modern PCIe PHY now have the ability to serve interrupts when down-training occurs much like they would for AER errors or hard link failures. Does FB use their own silicon in these data centers? Having this feature enabled is crucial when you get up to gen4 speeds. Weirdly I don’t see any detail about the gen used here though.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#9Earlier quoted context omitted.
Aren’t pciutils dependent on hardware support though? I have a couple of Asus mobos that have weird mouse movement depending on how slightly angled the GPU is in the slot - I tried to use pciutils to diagnose the issue but the “gamer”-grade Core i7 I have in there wasn’t on support-list because Intel only enables PCIe diags on i9/X and Xeon chips. ...and I know a few DCs that loaded up on cheap low-grade i3/i5 chips…
...Why would sysfs expose something that your hardware doesn't support while pciutils doesn't?
I assumed sysfs exposes a subset of available information: that pciutils can expose more data given hardware support. I know some of the more basic pciutils ran on my computer, though.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#10Earlier quoted context omitted.
...Why would sysfs expose something that your hardware doesn't support while pciutils doesn't?
I assumed sysfs exposes a subset of available information: that pciutils can expose more data given hardware support. I know some of the more basic pciutils ran on my computer, though.
FWIW, pciutils can use a variety of different access methods. Use lspci -A help to see the different access methods, and then lspci -A $method to use a different access method. Note that one of the access methods on linux is linux-sysfs, so pciutils can use sysfs under the hood.