Live data from Hacker News

How Facebook deals with PCIe faults to keep its data centers running reliably

engineering.fb.com

31–40 of 53 posts

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#31
post #12

Earlier quoted context omitted.

I believe this is called "technical marketing" Does Facebook still want to launch an own AWS clone?

Not sure if they did who would end up being the bigger asshole. My opinion of AWS is already at an all time low, but my opinion of Facebook has always been low too. At least it would introduce some competition for AWS.

AWS all time low? What planet are you getting your insights from? Because down here on earth they are making billions of dollar. AWS is an absolute juggernaut.

AWS and we as users get an insanely well made product. I just delivered an analytics solution built with AWS components and its frigging great to work with. Maybe with one exception; IAM.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#32
post #5

Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.

This is likely very deliberate. Publishing error rates of vendor equipment could violate NDAs or just damage vendor relations. The number of units purchased can be used by stock market analysts to predict FB growth or vendor sales, so the PR/legal team might prevent the publishing of details which could be used to that effect. Those are my guesses as to why actual numbers would be left out.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#33
post #28

Earlier quoted context omitted.

I know many people who won't dare touch GCP with a ten foot pole simply because they're afraid of Google's random banning AI wiping their digital lives. Azure is a minefield that's only worth it when on a .NET stack.

Re Azure, what makes you say that? Can you be specific?

Recently I tried to set up some pretty basic Office 365 stuff, and in the end I had to pay someone from Upwork to run a bunch of PowerShell commands because the admin UI simply ignore or timed out when I tried to set stuff. Between that, the absolutely dreadfully confusing product URLs and so on, the chances of me ever trusting MS enough to run servers for me is slim.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#35
post #28

Earlier quoted context omitted.

I know many people who won't dare touch GCP with a ten foot pole simply because they're afraid of Google's random banning AI wiping their digital lives. Azure is a minefield that's only worth it when on a .NET stack.

Re Azure, what makes you say that? Can you be specific?

I recently built a bunch of stuff on Azure, and the product limitations are absolutely insane. I came up with a new term in the aftermath of this project: "Almost Minimal Viable Product" (AMVP). It's like an MVP, but not quite.

Just in the last few weeks I hit these fun "broken by design" issues:

Availability Sets decrease your availability because they force big-bang changes for the member VMs. They flat out prevent one-VM-at-a-time changes for large categories of settings, such as SKU family, Accelerated Networking, and Proximity Placement groups. VMware had similar features yet no such limits over a decade ago.

Speaking of availability sets, you can create one with the number of fault domains set to "1", which makes sure that your critical servers are all plugged in to the same power rail and will fail together, ensuring disaster. You can't change this parameter.

Oh don't worry, their doco helpfully tells you to work around these glaring issues by deleting the VMs and recreating them. Except that this wipes out a bunch of settings and data that can't be recreated. Data loss is their official solution!

Speaking of data loss: You can't move a VM from one Recovery Vault to another without permanently deleting its backups first.

Other than that, Recovery Vault is a great product with only a few small feature gaps, such as the inability to back up Ultra SSD disks. You know: the type used for the most important VMs!

They NAT IPv6. I still can't get over that. You can't do anything if you enable IPv6 anywhere. For example, they just released Virtual WAN, but it has exactly zero support for IPv6. It just flat refuses to work with it. Ditto for NAT Gateway, which will refuse to NAT IPv4 if you have IPv6 enabled.

Speaking of IPv6: They generously hand them out in blocks as large as 16 addresses at a time. You get a whole /124 range all to yourself!

Stopping a VM can take up to half an hour, sometimes 2-3 hours. I hope you weren't making those aforementioned big-bang changes!

They have Gen 2 images for Windows, but not Windows + SQL Server. In fact, SQL Server has a random subset of the images you'd expect it to have, with gaps all over the place.

You can enable OS-level ("Guest") metrics, but you can only see them one VM at a time, not in any multi-VM view. You cannot imagine how fiddly this is to enable through any kind of automation.

Recently, Log Analytics randomly stopped collecting IIS logs world wide. The fix is to restart the service manually. This went on for like a week.

Some of their managed certificates are validated based on the "TLD" name, not the DNS zone name. So if you have "dev.myapp.dept.org.megacorp.com", then you have to figure out who receives these emails at the head office. In a different time zone. PS: They've never heard of you, and this looks 100% like a phishing attempt. PPS: This is totally broken for some domains, it goes to the wrong one by design.

Look, I could go on, but listing all of the showstopper issues I encountered while doing rather trivial stuff in just the last few weeks would require several hours of typing, and I'm tired because I was up until 9:30pm waiting for Azure VMs to take their sweet time to reboot.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#36
post #2

Pcicrawler looks interesting, but its too bad that it uses sysfs, making it tied to Linux. I wonder how hard it would be to make it use pciutls, which would make it portable to FreeBSD, MacOS, Windows, etc.. EDIT: There is a libpci for python that uses pciutls...

Even though it seems to only officially support CentOS, I gave it a shot on a GCP Ubuntu instance. `pcicrawler` output some info, but `pcicrawler -t` was just blank.

Seems similar to `lstopo`

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#37
post #5

Is anybody else frustrated that this somewhat lengthy article was relentlessly vague with the actual numbers? It was so consistent it felt like something you would present to a person allergic to technical detail.

This is likely very deliberate. Publishing error rates of vendor equipment could violate NDAs or just damage vendor relations. The number of units purchased can be used by stock market analysts to predict FB growth or vendor sales, so the PR/legal team might prevent the publishing of details which could be used to that effect. Those are my guesses as to why actual numbers would be left out.

One of the many reasons that I love Backblaze as a cloud backup solution is that they publish these numbers regularly. If there are bad drives in the market place, we as consumers have a much better view of it than Amazon reviews et al.

https://www.backblaze.com/b2/hard-drive-test-data.html

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#38
post #28

Earlier quoted context omitted.

I know many people who won't dare touch GCP with a ten foot pole simply because they're afraid of Google's random banning AI wiping their digital lives. Azure is a minefield that's only worth it when on a .NET stack.

Re Azure, what makes you say that? Can you be specific?

I had to spend literal weeks back and forth with Microsoft for getting my MPN account running - the end result was to go somewhere deep in PowerBI to unlock some random DNS setting. No I'm not kidding - I have kept all that stuff in my Twitter DMs with their support. Utter nightmare.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#39
post #24

Earlier quoted context omitted.

Not sure if they did who would end up being the bigger asshole. My opinion of AWS is already at an all time low, but my opinion of Facebook has always been low too. At least it would introduce some competition for AWS.

Some competition for AWS? Seems like Azure and GCP are competing pretty hard don't you think?

Until there's something like CDK for Azure and GCP I am hesitant to try it. Being able to leverage a mature language (TypeScript)'s completions and type system to code up an infra is absolutely wonderful. The docs are lacking a lot of examples, mind.

Re: How Facebook deals with PCIe faults to keep its data centers running reliably

#40

Earlier quoted context omitted.

Not sure if they did who would end up being the bigger asshole. My opinion of AWS is already at an all time low, but my opinion of Facebook has always been low too. At least it would introduce some competition for AWS.

AWS all time low? What planet are you getting your insights from? Because down here on earth they are making billions of dollar. AWS is an absolute juggernaut. AWS and we as users get an insanely well made product. I just delivered an analytics solution built with AWS components and its frigging great to work with. Maybe with one exception; IAM.

Meanwhile I'm sitting here after 4 visits to every dashboard I can find deleting and turning things off, and I still can't figure out where the fuckers are charging me from, because they don't care to improve that.
Post reply on HN