Live data from Hacker News

WhatsApp scaled to 1B users with only 50 engineers

quastor.org

321–330 of 462 posts

Re: WhatsApp scaled to 1B users with only 50 engineers

#321
post #290

Earlier quoted context omitted.

It’s comical that all of the replies to what you’ve written claim you’re wrong, because you’re actually 100% right. And this is even more true when you remember that the company had fewer than 20 people for a long time, and that there was only one real goal , that seems to be forgotten: they needed to clone BlackBerry Messenger and get it working on every platform possible. That was it. That’s all they needed to do,…

Nice story. What are you doing now?

Post-FAANG digital infrastructure. This is that moment, when the cycle starts all over again.

Re: WhatsApp scaled to 1B users with only 50 engineers

#322
post #124

There was a joke, or maybe anecdote, going around the Israeli high tech industry when Google purchased Waze When it came time to get an overview of the codebase being purchased. The Google Maps car navigation iOS team had over 200 engineers. The Waze iOS team had 2

> The Google Maps car navigation iOS team had over 200 engineers.

I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many.

Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.

Re: WhatsApp scaled to 1B users with only 50 engineers

#323
WhatsApp has to be one of those most easily scalable projects to work on. You have the freedom to divide chatrooms across different databases/shards. Same goes for notifications, where users can be divided since they're all isolated. The hardest part is just coordinating all the moving pieces in a way that can grow well, and that mostly just involves avoiding pitfalls in your design that might bottleneck.

Re: WhatsApp scaled to 1B users with only 50 engineers

#324
post #130
post #107

Earlier quoted context omitted.

WhatsApp was built on Erlang and FreeBSD, which feels way more exotic to me

FreeBSD has been released in 1993 and has been a well known OS in the hosting world. Erlang has been released in 1986 and has been a well known language for message systems. When I think about exotic I think about some new tech that sound really nice but is still very unproven.

JavaScript is from 1996, it's 25 years old, and apps built with it are used by billions. With regards to the backend, it has a vibrant full stack ecosystem, it's performant, simple, and easy to hire talent for.

It's okay not to enjoy JavaScript (I'd always prefer to go with Dart or Rust myself), it's okay thinking that other languages (and ecosystems) are better equipped to solve some problems, but calling it unproven and exotic is very surprising to me.

Re: WhatsApp scaled to 1B users with only 50 engineers

#325

My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out. Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers. WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long),…

>WhatsApp success is not about what WhatsApp did, it's about everything they didn't do.

I wrote something similar [1] about its success not long ago on HN,

>WhatApp grew because it was the only IM that worked on Smartphone. Thanks to Erlang. Yes. ICQ didn't bother. Microsoft refuse to accept Smartphone is the future of computing hence there never was MSN on Smartphone. Other contender tried but none of them managed to Scale. I still remember all my colleagues and friends used to download instant messenger on our phone to test them out, creating a group and basically DDoS each other until the system or network crash. No IM was able to handle the traffic load, not only WhatsApp did it many times better than their competitors, they also had almost all platform support, from Symbian to Blackberry.

There is another part of the story that dont get much appreciation, is how they managed 2 million connection per box on a what is now a relatively slow server. The latest M1 Max would have been a lot faster. I often wonder if they could have pushed to 10 Million connection per box with same latency under current CPU and Memory. ( Not saying it is a good idea. )

[1] https://news.ycombinator.com/item?id=25984924

Re: WhatsApp scaled to 1B users with only 50 engineers

#326
post #321

Earlier quoted context omitted.

Nice story. What are you doing now?

Post-FAANG digital infrastructure. This is that moment, when the cycle starts all over again.

Anything of note we humble readers should take a look at?

Re: WhatsApp scaled to 1B users with only 50 engineers

#327
post #290

My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out. Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers. WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long),…

It’s comical that all of the replies to what you’ve written claim you’re wrong, because you’re actually 100% right. And this is even more true when you remember that the company had fewer than 20 people for a long time, and that there was only one real goal , that seems to be forgotten: they needed to clone BlackBerry Messenger and get it working on every platform possible. That was it. That’s all they needed to do,…

Also can't overlook that for many SMS was ludicrously overpriced, but the early basic smartphones (palm and symbian) often had wifi capability; WhatsApp provided secure, ludicrously cheap instant messaging across all meaningful handsets.

And that's all it had to do; the telecoms weren't interested in making SMS cheap or secure, RIM didn't want BBM on anything but their devices, ICQ/AIM/MSN/etc were similarly unavailable.

Re: WhatsApp scaled to 1B users with only 50 engineers

#328

It crossed my mind while considering this that a figure like this doesn’t mean the same as it would have pre-cloud infrastructure. My team is fairly experienced but not so much in infrastructure. Even so, we build and deploy on Google cloud with fairly excellent scalability and ease. Ten years or longer ago I suspect the story would be different - the infrastructure and scale constraints could have pinned us at times…

> And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for.

I was at WhatsApp from 2011 to 2019.

Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway.

Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno.

We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource).

What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly?

We self-hosted our code repository and bug tracker.

Re: WhatsApp scaled to 1B users with only 50 engineers

#330
post #304

My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out. Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers. WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long),…

> They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step This is the key mistake. Having teams of 2-3 with a manager each is way too many managers. If you have a clear focus, small teams are great because they can work autonomously, with little management and good results. You can't hire your way out of coordination hell. I w…

Post acquisition WhatsApp had to start making money.

For all the doom and gloom Facebook is still too scared of putting ads in there.

Post reply on HN