Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Facebook deals with PCIe faults to keep its data centers running reliably

engineering.fb.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#11

Post by [deleted] »

DaiPlusPlus wrote: post #4
drewg123 wrote: post #2

Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc.. EDIT: There is a libpci for python that uses pciutls...

Aren’t pciutils dependent on hardware support though? I have a couple of Asus mobos that have weird mouse movement depending on how slightly angled the GPU is in the slot - I tried to use pciutils to diagnose the issue but the “gamer”-grade Core i7 I have in there wasn’t on support-list because Intel only enables PCIe diags on i9/X and Xeon chips. ...and I know a few DCs that loaded up on cheap low-grade i3/i5 chips…

[deleted]

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#12

Post by baybal2 »

jandrese wrote: post #5

Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.

I believe this is called "technical marketing"

Does Facebook still want to launch an own AWS clone?

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#13

Post by nautilus12 »

baybal2 wrote: post #12
jandrese wrote: post #5

Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.

I believe this is called "technical marketing" Does Facebook still want to launch an own AWS clone?

Not sure if they did who would end up being the bigger asshole. My opinion of AWS is already at an all time low, but my opinion of Facebook has always been low too. At least it would introduce some competition for AWS.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#15

Post by maccam94 »

th33ngineer wrote: post #8

> Bad…link speed…and bad…link width…were other concerning PCIe faults. These faults can be difficult to detect without some sort of automated tool… Most modern PCIe PHY now have the ability to serve interrupts when down-training occurs much like they would for AER errors or hard link failures. Does FB use their own silicon in these data centers? Having this feature enabled is crucial when you get up to gen4 speeds. W…

FB doesn't want hardware to run at lower than rated speeds. Their tool allows them to detect when it happens and remediate the issue.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#17

Post by numpad0 »

maccam94 wrote: post #15
th33ngineer wrote: post #8

> Bad…link speed…and bad…link width…were other concerning PCIe faults. These faults can be difficult to detect without some sort of automated tool… Most modern PCIe PHY now have the ability to serve interrupts when down-training occurs much like they would for AER errors or hard link failures. Does FB use their own silicon in these data centers? Having this feature enabled is crucial when you get up to gen4 speeds. W…

FB doesn't want hardware to run at lower than rated speeds. Their tool allows them to detect when it happens and remediate the issue.

OP claims it shouldn’t be “difficult to detect (...) because the hardware is working” because most commercially sold host controller chips would generate interrupt and report errors, unless Facebook is using something nonstandard that don’t.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#20

Post by maccam94 »

numpad0 wrote: post #17
maccam94 wrote: post #15

Earlier quoted context omitted.

FB doesn't want hardware to run at lower than rated speeds. Their tool allows them to detect when it happens and remediate the issue.

OP claims it shouldn’t be “difficult to detect (...) because the hardware is working” because most commercially sold host controller chips would generate interrupt and report errors, unless Facebook is using something nonstandard that don’t.

The hardware is reporting the errors to the kernel but not crashing the system. It's "difficult to detect" because unless you are specifically monitoring for those stats, the only issue you'll see is degraded performance on an occasional machine (assuming you are watching carefully enough to even discern the performance delta). Some of the error counters are even predictive of an issue rather than something that is actively impacting performance. The FB software is basically scraping those messages and bus stats into JSON that can be consumed by their monitoring infrastructure.
Post reply on HN