WhatsApp seems to be a great example of designing your system with the best tools available. To go from nothing to pretty much being the de-facto modern replacement for SMS in a couple of years is an amazing achievement. I really need to sit down and spend some time properly learning a functional language. I can hack my way through Haskell code but I do it in a very non-Haskell way. I really need to change that. My p…
Why WhatsApp Only Needs 50 Engineers for Its 900M Users
121–130 of 255 posts
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#122Comments in this thread are quite cringeworthy. People calling WhatsApp a simple application seem to fail to understand the cost of simplicity under such scale.
It wasn't a great article from wired. I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. Yes, it's an eclectic choice, but I find systems/component architecture much more interesting than language and program architecture in scaling considerations. A bit more info here: http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... With regard to fun ch…
It's here https://www.reddit.com/r/programming/comments/3l2spe/why_wha...
In its entirety:
I can explain a bit more in depth than the author did.
Erlang uses a model known as the actor model. This means that each part of your program is a completely independent entity like a mailbox, known as an actor. For one actor to communicate with another actor, it simply puts a message in the intended actor's mailbox. Since each actor is independent, this means that we can spawn as many actors as we want to read messages from a single mailbox, and do what is requested.
In terms of what WhatsApp is doing, this means that they can have (potentially infinite) conversations going on between users all at the same time. If any part of their pipeline is working too slowly (there are more messages being put in the mailbox than taken out) they can just spawn additional actors to read mail (at any time they want, even programmatically). Additionally, as alluded to by the author, in Erlang you can do something known as "hot loading," which means spawning newer versions of actors (when the implementation changes) while still running the old actors so that there is never any loss of service.
Haskell is similar to Erlang in its ability to scale. In fact, there is a library in Haskell known as HdpH-RS that enables fault-tolerant (the programmer does not have to worry about intermediate failure at all) distributed computing/memory (tested up to 1400 cores). The main reason for this is that Haskell is strictly-typed, meaning all types must be defined (or derivable) at compile time, and immutable, meaning that variables cannot change their values after they are defined.
Due to all of this strictness, you never have to worry about errors found in languages like C such as segfaults, or improper use of a function since you can enforce usage with types. For instance a function that should only accept unit vectors would use a type UnitVector instead of Vector (Note that this approach is generally a bad idea in C/C++/Java etc). Essentially, if your program compiles, then it will run correctly (generally the only real errors you run into are logical ones -- the algorithm itself being incorrect). This makes it really easy to refactor code later on. The reason Facebook can handle urgent tasks so quickly is mostly due to Haskell's generics, which are like templates from C++ or generics in Java. The main difference however is that Haskell lets you create generic mixins for types known as type classes. This means that you can write extremely general code to handle almost anything in the high level. Then, whenever you need to handle something new, you just have to choose which mixins you want and just add them to your type by defining the interface. For example, I could make a Multipliable type class, I would require the programmer to only define the multiplication operator for his type, and then it would automatically create things like pow or sqr.
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#123> Why WhatsApp Only Needs 50 Engineers for Its 900M Users Answer: because sending short messages from A to B is basically a solved problem. There is even a programming language (Erlang) that was made with this application in mind. The prototypical "Hello World" example for Erlang is a messaging application.
Yep. And of course, Erlang is really only a functional language in the small , the parts that don't matter so much. In the large , arguably the scale that matters, it's an actor language, with highly encapsulated entities exchanging messages. So essentially what Alan Kay had in mind when he coined the term "object oriented". http://gagne.homedns.org/~tgagne/contrib/EarlyHistoryST.html
Also, let's not forget to give credit to Carl Hewitt who came up with actors in the first place [2].
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#124Comments in this thread are quite cringeworthy. People calling WhatsApp a simple application seem to fail to understand the cost of simplicity under such scale.
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#125> Why WhatsApp Only Needs 50 Engineers for Its 900M Users Answer: because sending short messages from A to B is basically a solved problem. There is even a programming language (Erlang) that was made with this application in mind. The prototypical "Hello World" example for Erlang is a messaging application.
Sure, the initial business case is "Send a message from A to B". But it's not hard to think of more things to do: - What if A has a Nokia and B has an iPhone? - What if you want A, B, and C to have a chatroom? - What if someone starts spamming random people? - How do we QA the thing for all these platforms? - How about someone to keep the visual design fresh? - What about a million corner cases like when someone is n…
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#126Earlier quoted context omitted.
I believe that the line of reasoning of your parent still has merit. Erlang OTP was developed for telephony systems and has proven to be very reliable for distributed system. Messaging maps relatively nicely to Erlang OTP and as such Whatsapp are leveraging decades of research and work by Ericsson. Of course, this is how it should be: a good software infrastructure should save users time. Obviously, creating Whatsapp…
I think that indeed the important thing here. The article try to focus a bit too much on some intrinsic qualities of the language as if it was the silver bullet of scalability while the real lesson is that they used the right tool for the job. Not so long ago, maintaining hardware that could support 900M user would have required an IBM scale company. A single developer today can develop, deploy and maintain a medium…
* SoftLayer is their cloud provider, bare metal machines, fairly isolated within the network, dual datacenter configuration
Softlayer is an IBM Company, so ultimately it DOES require an IBM scale company to run this ( at some point in the chain).
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#127Earlier quoted context omitted.
Sure, the initial business case is "Send a message from A to B". But it's not hard to think of more things to do: - What if A has a Nokia and B has an iPhone? - What if you want A, B, and C to have a chatroom? - What if someone starts spamming random people? - How do we QA the thing for all these platforms? - How about someone to keep the visual design fresh? - What about a million corner cases like when someone is n…
Pretty much every single one of your test cases has already been solved in the XMPP specification, which WhatsApp is likely using (essentially all modern chat services/servers/clients speak XMPP).
It's easy to see what pieces you need. But there's still work in glueing them together and testing the result.
I think 50 is a sensible number of in-house engineers for this sort of thing. You want cover as well in case someone leaves or gets ill. You want people to be able to upgrade the system when things change. You want people to look ahead at future requirements. And you want to be able to do these things all at once, with slack built in for peak load.
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#128Earlier quoted context omitted.
I'm sure you could do it in an afternoon, right? Just like anyone on HN can make a "basic CRUD app" like Facebook in a weekend. I really dislike the arrogant attitude behind these types of comments. Have you worked at WhatsApp? Do you have any idea what they spend their days doing, what their systems look like, what their requirements are like? WhatsApp have close to a billion users, 50 engineers and they manage to p…
Have you worked at WhatsApp? Do you have any idea what they spend their days doing, what their systems look like, what their requirements are like? Do you? If yes, please enlighten us. Otherwise I don't see the point of your objection. No one said that it can be done in a weekend. All that the parent comment said is that the fundamentals of building WhatsApp are already solved. He/she didn't dismiss it. Just because…
Go watch this: https://www.youtube.com/watch?v=TneLO5TdW_M#
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#129WhatsApp seems to be a great example of designing your system with the best tools available. To go from nothing to pretty much being the de-facto modern replacement for SMS in a couple of years is an amazing achievement. I really need to sit down and spend some time properly learning a functional language. I can hack my way through Haskell code but I do it in a very non-Haskell way. I really need to change that. My p…
https://pragprog.com/book/elixir/programming-elixir <-- dave thomas has a great writing style.
Re: Why WhatsApp Only Needs 50 Engineers for Its 900M Users
#130Earlier quoted context omitted.
It wasn't a great article from wired. I feel the language choice is probably the least contributing factor in terms of what's interesting at that scale. Yes, it's an eclectic choice, but I find systems/component architecture much more interesting than language and program architecture in scaling considerations. A bit more info here: http://highscalability.com/blog/2014/3/31/how-whatsapp-grew-... With regard to fun ch…
I agree, Wired didn't cover the subject with the kind of detail programmers would expect.
Wired isn't writing articles for programmers. It's a pop-science magazine writing tech articles for the mainstream.