Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta’s Hyperscale Infrastructure: Overview and Insights

cacm.acm.org

Post reply on HN
Print view View original

61–70 of 182 posts

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#61

Post by sofixa »

udev4096 wrote: post #36

As much as I hate clownfare, it would be very interesting if they published something like that because of the sheer number of data centers they operate

They have published a lot of information on their blog[1]. It's piecemeal, and around other articles about optimisations or security fixes or failures/postmortems, but it's there. Stuff like:

https://blog.cloudflare.com/how-we-use-hashicorp-nomad/

https://blog.cloudflare.com/cloudflare-deployment-in-guam/

https://blog.cloudflare.com/behind-the-scenes-with-stream-li...

1 - https://blog.cloudflare.com/

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#62

Post by sofixa »

ribadeo wrote: post #32

At least half their gak is due to them NOT moving quickly and NOT wanting to break things. IIRC, graphql is a means of papering over a bunch of legacy APIs. They removed foreign keys from mysql using it as a column store db, a vestige of the original LAMP stack still on PHP. I don't think Meta infrastructural choices are applicable to most folk. What does serverless land your average dev? A high AWS bill. Elastic man…

> Did you know that you can use YAML and provision actual cloud provider resources with boring tech? Welcome to Ansible

Anyone using Ansible for cloud infrastructure management is not to be taken seriously. It's among the worst tools for the job - not (always) idempotent, no state tracking, slow, very limited in the resources it can manage, very lacking templating, fun stuff like "state: absent", running, and then having to remove the corresponding lines to delete, etc etc. You're literally better of bash scripting the cloud provider's CLI than using Ansible. Terraform/OpenTofu, Pulumi/tfcdk if you hate your future self are just clearly so much better.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#64

Post by javier2 »

vasco wrote: post #4

> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product te…

They already had these teams and they already had the infrastructure. 100M is a drop in the ocean to Meta

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#65

Post by dijit »

lionkor wrote: post #54

> the image is not cached at CDN109 when the user requests it, CDN109 forwards the request to a nearby PoP. The PoP then forwards the request to the load balancer in a datacenter region, which retrieves the image from the storage system. Say I want a 1MB image, wouldn't it be faster to serve me the 1MB image over a slow connection with 100ms latency, than going through multiple hops of increasing latency, with multip…

DC-to-DC networking is typically much higher throughput, and often lower latency (due to fewer hops).

Consider also that the CDN can fetch the whole image async after the first request, but the nature of TCP (that you'll be using to fetch objects) is serial, meaning you can only fetch about 1k each round-trip.

Now, in reality the situation is way more complex, because various hacks have been added over the years, for example your browser might grab the size header and just create a bunch of download threads for your image with various offsets, the TCP scaling window might come into play; but largely most people set 1500b for their MTU on consumer hardware, and there's some overhead from TCP/IP and so on.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#66

Post by eloisius »

pinoy420 wrote: post #44
vasco wrote: post #4

> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product te…

Sounds like a horrific place to work. Imagine the pressure

I’d far rather work in an environment that required and enabled me to get shit done than the opposite. My last W-2 job required a quarter-long planning period and then I had to go from team to team pleading my case and begging them not to stone wall me. Each time requiring a specially tailored slide deck, going behind the scenes to ask what questions _that one guy_ is going to try and derail us with so I could have an answer ready, and often follow up with a second or third meeting to address concerns of the DBA team on what effect using a WHERE clause will have.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#67

Post by KaiserPro »

davedx wrote: post #3

Very interesting, in particular the explicit comparisons with hyperscalers. I almost wonder if this is preparation for them launching their own public cloud. Anyone from Meta care to comment?

No.

The infra is not isolated enough to be anywhere near ready for public offerings.

Sure there is process isolation, and its hard to break out of the jail/container/whatever you want to call the unit of execution. But its just not ready for public consumption.

The methods for monitoring, creating, deploying and scaling the "service(s)" are just too intertwined with internal access. Whilst there is sorta fine grained control on which other services you have access to, its nowhere like AWS et al.

the _other_ thing is that everything needs to be compiled to fit the platform. Its not run in VMs, its bare metal processes, with some rather fancy shims to isolate away libc (don't ask me more, I know that its there and some of the reasons why, but the mechanics are a mystery to me).

That platform that you compile to is a movingish target.

So no, meta isn't going to host thirdparty stuff, mainly because meta doesn't really have enough capacity for what it want to do now, let alone add more consumers.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#68

Post by WJW »

Simon_O_Rourke wrote: post #28
vasco wrote: post #4

> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product te…

They might have launched it quickly, but nobody (relatively speaking) gave a tinkers damn about the end product. All form and no function.

Form and no function is the essence of social media apps though, so they nailed that. 300 million monthly and 100m daily users is also pretty decent, so I don't think "nobody" cares. For comparison, that's approximately the same number of MAU as twitter.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#69

Post by average_r_user »

eloisius wrote: post #66
pinoy420 wrote: post #44

Earlier quoted context omitted.

Sounds like a horrific place to work. Imagine the pressure

I’d far rather work in an environment that required and enabled me to get shit done than the opposite. My last W-2 job required a quarter-long planning period and then I had to go from team to team pleading my case and begging them not to stone wall me. Each time requiring a specially tailored slide deck, going behind the scenes to ask what questions _that one guy_ is going to try and derail us with so I could have a…

The constant need for tailored slide decks is what's destroying my motivation atm

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#70

Post by _kb »

asdasd1234 wrote: post #8

Is there any public information available for the deploy/observability tool?

I assume it's changes since then, but this is from 2021: https://atscaleconference.com/conveyor-continuous-deployment....

There's also a good paper on the approaches to configuration management here: https://research.facebook.com/publications/holistic-configur....

Post reply on HN