I'm calling BS on this. Total lack of any specifics, and I don't need to know what brand of hardware is giving them fits. I've been responsible for tens of millions of dollars in hardware over the years of my career and PCIe has never been the source of my headaches. Disk drives, memory, and firmware bugs are the usual suspects. They can get right out of here with their holier-than-thou attitude on this.
How Facebook deals with PCIe faults to keep its data centers running reliably
41–50 of 53 posts
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#42Earlier quoted context omitted.
Some competition for AWS? Seems like Azure and GCP are competing pretty hard don't you think?
Until there's something like CDK for Azure and GCP I am hesitant to try it. Being able to leverage a mature language (TypeScript)'s completions and type system to code up an infra is absolutely wonderful. The docs are lacking a lot of examples, mind.
Pulumi?
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#43Earlier quoted context omitted.
Not sure if they did who would end up being the bigger asshole. My opinion of AWS is already at an all time low, but my opinion of Facebook has always been low too. At least it would introduce some competition for AWS.
AWS all time low? What planet are you getting your insights from? Because down here on earth they are making billions of dollar. AWS is an absolute juggernaut. AWS and we as users get an insanely well made product. I just delivered an analytics solution built with AWS components and its frigging great to work with. Maybe with one exception; IAM.
I'm talking about their moral/ethical compas.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#44Earlier quoted context omitted.
Some competition for AWS? Seems like Azure and GCP are competing pretty hard don't you think?
I know many people who won't dare touch GCP with a ten foot pole simply because they're afraid of Google's random banning AI wiping their digital lives. Azure is a minefield that's only worth it when on a .NET stack.
For any companies in retail, using AWS can be a hard sell because of Amazon being a competitor. GCP/Azure are much easier for them.
As for Google banning accounts, that is not a thing past a certain level. They would be in breach of contract – past a certain point the business has a contract, it's not a personal GSuite account paying by credit card, it's a business account with a more substantial contract, SLAs, etc.
Turning off a whole service for a company violating terms of service (like what happened with AWS+Parler) is another case.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#45I'm calling BS on this. Total lack of any specifics, and I don't need to know what brand of hardware is giving them fits. I've been responsible for tens of millions of dollars in hardware over the years of my career and PCIe has never been the source of my headaches. Disk drives, memory, and firmware bugs are the usual suspects. They can get right out of here with their holier-than-thou attitude on this.
I worked at a large cluster computing company and we did occasionally, very occasionally see PCIe problems. Note that a lot of people are now exporting PCIe over a cable, not just plugging into the mainboard, and that can be a source of problems ('oops, the PCIe cable was routed in a location that made it experience more EMF, vibration, and physical damage and then it started to show more errors).
These sorts of problems mainly show up if you're running your own fleet, designing your own servers (poorly/aggressively) and have a budget of $10B.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#46Earlier quoted context omitted.
AWS all time low? What planet are you getting your insights from? Because down here on earth they are making billions of dollar. AWS is an absolute juggernaut. AWS and we as users get an insanely well made product. I just delivered an analytics solution built with AWS components and its frigging great to work with. Maybe with one exception; IAM.
I'm talking about their moral/ethical compas.
Do they lie, cheat, steal, or could you elaborate on problems with their moral/ethical compass?
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#47I'm calling BS on this. Total lack of any specifics, and I don't need to know what brand of hardware is giving them fits. I've been responsible for tens of millions of dollars in hardware over the years of my career and PCIe has never been the source of my headaches. Disk drives, memory, and firmware bugs are the usual suspects. They can get right out of here with their holier-than-thou attitude on this.
I don't mean to be rude, but did you ever look? Swapping parts because something is wrong would fix weird PCIe errors as well.
Also, PCIe usage has probably grown exponentially over the past 10 years. Its possible facebook has an order (or two) more of PCIe "links" in their datacenters today compared to 10 years ago.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#48Earlier quoted context omitted.
Some competition for AWS? Seems like Azure and GCP are competing pretty hard don't you think?
Until there's something like CDK for Azure and GCP I am hesitant to try it. Being able to leverage a mature language (TypeScript)'s completions and type system to code up an infra is absolutely wonderful. The docs are lacking a lot of examples, mind.
Cloudformation is a better tool for infra.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#49Earlier quoted context omitted.
Re Azure, what makes you say that? Can you be specific?
I had to spend literal weeks back and forth with Microsoft for getting my MPN account running - the end result was to go somewhere deep in PowerBI to unlock some random DNS setting. No I'm not kidding - I have kept all that stuff in my Twitter DMs with their support. Utter nightmare .
What is MPN? That's not a core product or service I recognize.
Re: How Facebook deals with PCIe faults to keep its data centers running reliably
#50Earlier quoted context omitted.
Re Azure, what makes you say that? Can you be specific?
I recently built a bunch of stuff on Azure, and the product limitations are absolutely insane. I came up with a new term in the aftermath of this project: "Almost Minimal Viable Product" (AMVP). It's like an MVP, but not quite . Just in the last few weeks I hit these fun "broken by design" issues: Availability Sets decrease your availability because they force big-bang changes for the member VMs. They flat out preven…
That sounds painful indeed. I've never had to use any of these services or features on any other cloud so I can't compare. I've certainly heard that Azure Networking isn't great, but then again I am not someone who ever has needs that can't be met by what is being offered.
It sounds like you mostly deal with the Infrastructure level services - VMs, availability sets and networking.
What are your thoughts on the PaaS offerings (and there are many - too many to the point it gets confusing)? The Log Analytics issue seems very surprising - definitely something I'd expect to recover quickly without the need for intervention.