Live data from Hacker News

Meta’s Hyperscale Infrastructure: Overview and Insights

cacm.acm.org

11–20 of 182 posts

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#11
post #9
post #7

Earlier quoted context omitted.

I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.

What is threads lacking from a product perspective?

Novelty

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#12
post #9
post #7

Earlier quoted context omitted.

I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.

What is threads lacking from a product perspective?

It was given no thought or advertisement or anything. It -feels- like an engineering marathon to make a safe space from Twitter.

I don't know that it's a bad effort, but it's one that rose and died seemingly the same day.

I feel like more time with a good product person would have given more thought to fit, advertising, release, and so on.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#14
post #3

Very interesting, in particular the explicit comparisons with hyperscalers. I almost wonder if this is preparation for them launching their own public cloud. Anyone from Meta care to comment?

I left before they were Meta, and maybe things have changed, but I don't think they have any intention of being a public cloud. Yes, they've got a lot of similar services as a public cloud, but there's a lot of opinionated choices that make sense for them that I think would be hard to convince customers to accept.

Their infrastructure is cloudy, but it's built around mostly a single customer and assumes the infrastructure software people and the application software people communicate deeply and continously. Running on a public cloud isn't that similar, at least as a small customer.

Could they pivot towards being a cloud service? Probably, but they'd need to do a lot of work to make their platform viable and to earn trust of potential customers, and they'd be entering a crowded market; there's already 6 S&P100 companies in Cloud (Amazon, Google, Microsoft, Oracle, IBM, Salesforce), and tons of smaller players.

IMHO, given their revenues and profit margins, there's no reason to do all the work it would take to offer cloud services too. Unless there's some opportunistic large customer deal made. They also might also need to renegotiate their content node agreements if they use them to serve cloud customer traffic, and that's a long process.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#15
> In a datacenter environment, we prefer centralized controllers over decentralized ones due to their simplicity and ability to make higher-quality decisions. In many cases, a hybrid approach—a centralized control plane combined with a decentralized data plane—provides the best of both worlds.

This approach appears to be one of the most optimal designs for software networking (service mesh) and for storage (database operations) for organizations with large server counts. I was surprised to see their IP networking to follow the same model, rather than primarily relying on BGP.

It was omitted in this paper, but I would expect for local caching to be used to reduce load on L7 routers and for improved latency for database queries. Clients can invalidate caches and perform another lookup to the service mesh after a reasonable timeout (100-500ms).

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#16
post #7
post #4

> Moreover, once it [Threads] was developed, the infrastructure teams were given only two day’s notice to prepare for its production launch. Most large organizations would take longer than two days just to draft a project plan involving dozens of interdependent teams, let alone execute it. At Meta, however, we quickly established war rooms across distributed sites, bringing together both infrastructure and product te…

I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.

What? It has more than 300M users.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#17

Is there any public information available for the deploy/observability tool?

Unfortunately I didn't see any code posted for Conveyor. The following USENIX paper is available: https://www.usenix.org/system/files/osdi23-grubic.pdf .

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#18
post #13

Not a word about Thrift, perhaps it was too low level for an infra overview, but I would have expected it to make some technical impact from a global perspective.

I wouldn't be surprised if they have switched to GRPC for improved performance. They mentioned that RPC libraries are centrally maintained in their monorepo; a migration from Thrift to GRPC might have taken less than 6 months.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#19
post #7

Earlier quoted context omitted.

I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.

What? It has more than 300M users.

Today, maybe? I haven't kept up.

I'm talking specifically about its launch.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#20
post #7

Earlier quoted context omitted.

I thought the same thing, but then realize that Threads was a huge flop. Is it really a skill to very quickly release a dud app? I don't know the answer to that. Bypassing bureaucracy seems like heaven, but it feels like it also bypassed the product folks entirely.

What? It has more than 300M users.

How many of those are bots?
Post reply on HN