Live data from Hacker News

Meta’s Hyperscale Infrastructure: Overview and Insights

cacm.acm.org

161–170 of 182 posts

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#161

Earlier quoted context omitted.

[flagged]

Ach. You seem full of (unrelated, unwarranted) ragefulness, and you stand ready to handwave away both scientific studies and actual user experience. Without, I see, providing any grounded counterpoints. As a bonus, anyone who disagrees with you is a ‘tool’ of sinister ‘bad actors’ Are you quite sure you’re on the right forum? That’s not generally how we do things here.

[dead]

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#162

Earlier quoted context omitted.

Not to burst your weird leftist-rightist libertarian bubble, but vaccine passports are actually a fairly normal thing in most of the world mate. You know, it's just about documenting what vaccinations you were receiving throughout your lifetime. Go outside and touch the grass, you sound very angry.

[flagged]

No mate, read your comments again- the only one projecting anything here is you. For my part, I was quite sure how to label you and it's based on the first two rows of what you wrote, which is a mix of leftist, right-wing and libertarian views. Now as others have also noticed, you seem to have some deep-seated psychological issues, most notable in your rumbling and unconnected thoughts. I mean this without malice: seek help.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#163
post #125

Earlier quoted context omitted.

Threads has 320 million monthly active users and 100 million daily. It’s certainly not a flop. It’s almost as big as X globally. There’s a tremendous locality bias around Meta’s products. Social media use is very localized by geography and age. So if all your friends stopped using Facebook ten years ago, you assume that’s probably true everywhere, while in fact they’ve added several billion users on FB since then.

I frankly don't believe those numbers at all. They've massively pumped it through integration with Instagram, and have integrated dark patterns around profile creation and follows based entirely on Instagram social networks. Yes, a lot of people do actively use Threads, but it's absolutely not a first class social network [yet].

I share the doubt here. I believe I have seen exactly one threads link in the wild (Excluding people listing all their social accounts).

If it's got that level of usage, and not just people clicking on the occasional recommendation from Facebook/Instagram, it's remarkably insular.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#164
post #88

Earlier quoted context omitted.

Sounds like a horrific place to work. Imagine the pressure

You get addicted to the adrenaline. If you’re with really good people that kind of pressure is actually fun. Sort of like how a championship sports game is fun.

I do not mean this in an offensive manner at all. Are you between the age of 18-30?

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#165

Earlier quoted context omitted.

Not to burst your weird leftist-rightist libertarian bubble, but vaccine passports are actually a fairly normal thing in most of the world mate. You know, it's just about documenting what vaccinations you were receiving throughout your lifetime. Go outside and touch the grass, you sound very angry.

[flagged]

... speaking of vaccines "not preventing transmission"...for the love of god, please learn how the vaccines work. Did they not teach you in the primary school that it's about priming body for the immune response by introducing a very small concentration of the virus into the body. No vaccine can prevents a transmission, it's purpose is to enable your body to respond and reduce the severity of the infection symptoms. Have the education standards really deteriorated so much, or is this just the effect of overexposure to the social media, I wonder?

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#166
post #158

Earlier quoted context omitted.

I frankly don't believe those numbers at all. They've massively pumped it through integration with Instagram, and have integrated dark patterns around profile creation and follows based entirely on Instagram social networks. Yes, a lot of people do actively use Threads, but it's absolutely not a first class social network [yet].

Honestly it’s more the case that X is not a first-tier social network anymore. It now plays in second league with the likes of Threads and Pinterest. It retains important bubbles like Silicon Valley VCs and media people, but globally it’s just not significant with its single-digit market share.

Twitter was the the social network where reporters and important people congregated.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#167

Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh.

>Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh.

If only which that were true, it is more like building our own digital prison.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#168

It's unfortunate there is so much cynicism and negativity in these comments. I understand many people here strongly dislike Meta but the actual article is awe inspiring to me. I didn't know how extensive and complicated the infrastructure underpinning the modern digital world was. Reading this article and seeing the sense of scale is mindblowing to me. It feels almost like a miracle and like magic. A "wonder of the w…

> I feel like if you took some old science fiction writers from back in the day and showed them this article, you would find sheer awe in their faces too. I can't tell if this is satire. The science fiction writers from back in the day wrote about humans exploring other planets, building sentient machines, encountering aliens or maybe developing psychic powers. (looking at you two, Philip K. Dick and Theodore Sturgeo…

I'm getting downvoted here but can't for the life of me guess why. If you've actually read Isaac Asimov and his peers, as I have, they wrote about robots with complex inner lives and space ships that traveled to alien civilizations. There is no reason whatsoever to think they would experience any more "sheer awe" at Meta's infrastructure than any other random person from their time period. I honestly think they would be disappointed that it was all done to sell advertisements to people rather than to help humanity explore the universe -- or anything else of actual value.

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#169

Amazing. All of this truly wild, impressive technology and some of the absolute best engineers in the world, just to shove more ads in people's eyeballs. Sigh.

The tragedy of our times is that some of our brightest minds are working tirelessly, not to cure diseases or to explore the stars, but to improve the probability that someone will click on an ad by 0.1%.

[deleted]

Re: Meta’s Hyperscale Infrastructure: Overview and Insights

#170
post #54

> the image is not cached at CDN109 when the user requests it, CDN109 forwards the request to a nearby PoP. The PoP then forwards the request to the load balancer in a datacenter region, which retrieves the image from the storage system. Say I want a 1MB image, wouldn't it be faster to serve me the 1MB image over a slow connection with 100ms latency, than going through multiple hops of increasing latency, with multip…

This is a reasonable question, but I think you've made some wrong assumptions.

Typically, the CDN nodes will be within your ISPs network, typically on the way to the PoP (or at the PoP). You're unlikely to see a significantly different round trip time if your request is proxied through the CDN vs going to the remote datacenter directly. But, even if your total round trip time is a little longer (10ms in your hypothetical), you get benefits from local TCP and/or TLS termination.

At ~ 100ms latency, your effective bandwidth is usually limited more by round trip than actual connection bandwidth, because of congestion control / slow start. A 1 MB image is going to be somewhere around 700 packets with ~ 1500 MTU. Assuming standard congestion control, the initial congestion window is 10 packets, for each packet acked, the server can send two packets. Assuming you and the server and the path between have infinite bandwidth and a large enough receive window, the client wound receive 10 packets at t=100ms after the initial request, then 20 additional (total 30) at t=200ms, ... 320 additional (630 total) at t=600ms, and the remainder at t=700ms. If your effective bandwidth is less than about 40 Mbps, you're going to hit congestion in the 6th bunch of packets, but any connection speed above that and your 1 MB transfer is going to take 700 ms. If you've got a much smaller MTU, you might need more round trips; and if you've got a larger MTU, you could end up with less round trips, but word on the street is inter-AS jumbo packets is rare, and Path MTU Detection isn't great, so lots of servers force an effective max MTU of 1500 or less, because it's easier to send smaller packets to everyone than to fix path mtu issues.

But if we have your steps of 10 ms to the CDN, 10ms to the PoP, 90 ms to the datacenter, and let's assume the transit time between you and the CDN and you and the PoP is symmetric, we get

Your request starts at t=0, CDN to PoP request starts at t=5 ms, PoP to DC request starts at t=10ms, then the PoP to DC request takes 7 round trips = 630 ms for all data, and you get that 10ms later at 640 ms. Your first byte of response is delayed by 10 ms, but it's worth while because time to last byte decreases by 60 ms. If the transit times between hops are asymmetric, the times that the client gets data don't change, but the math makes my head spin more.

If you change it up, and say you're 50 ms round trip to the CDN, which is 50 ms round trip to the PoP, which is 50 ms to the DC, first byte jumps to t=150 ms, but time to last byte would be 7 * 50 ms + 100 ms = 450 ms.

And, if the PoP -> DC connection happens over a warm socket, with an appropriate congestion window and receive window, the transfer from the DC to the PoP can happen much faster. Certainly, you could potentially have a warm connection to the data center from the client as well, but most services don't want to have millions or billions of client connections to all their datacenters, so running them through PoPs or CDNs can be pretty handy.

There's some handwaving here (I ignored processing time at each hop, but it's usually pretty low), but it's really helpful to process congestion control closer to the user. Any lost packets can be resent sooner for much quicker recovery if managed locally as well. And for things that are cachable, read-through caching with a CDN -> PoP -> Datacenter approach makes a lot of sense to reduce demand on the Datacenter, and benefiting from likely locality of reference --- people in the same area / on the same ISP are likely to fetch images that others in their area have fetched.

Post reply on HN