Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The C10M problem

c10m.robertgraham.com

Post reply on HN
Print view View original

91–100 of 116 posts

Re: The C10M problem

#91

Post by SkyMarshal »

alberth wrote: post #87

WhatsApp is achieving ~3M concurrent connections on a single node. [1][2] The architecture is FreeBSD and Erlang. It does make me wonder, and I've asked this question before [3], why can WhatsApp handle so much load per node when Twitter struggled for so many years (e.g. Fail Whale)? [1] http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2... [2, slide 16] http://www.erlang-factory.com/upload/presentations/55…

Haven't you answered your own question? FreeBSD + Erlang vs Rails. Not hating on Rails but it wasn't remotely designed for this use case, Erlang was and there's arguably nothing better in the world at it. Twitter got better after they moved to the JVM, another battle-hardened platform designed for scale.

Re: The C10M problem

#92

Post by erichocean »

justincormack wrote: post #65
wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

I think there has been a renewed interest around 10Gb ethernet (and SSD). Basically computers were grossly limited by IO with gigabit ethernet and hard drives, you could buy a pretty low end box and saturate them, instead people bought high end boxes and ran slow code. But your average slow code wont perform well on 10Gb.

Indeed. I've got an architecture right now that I could actually do everything I need to do in a single rack with 40GigE adaptors, except...I can't (easily) get that kind of bandwidth to so few boxes using third parties, and more importantly, I can't get the bandwidth cheaply that way.

So instead, I'm scaling out with mostly idle Intel E5-1620v2 4 core (8HT) boxes and single 1GbE connections because that's about 70% cheaper for access to the same bandwidth. In fact, bandwidth is actually the only cost we even monitor anymore, because that's what we're limited by (the machines come with far more bandwidth allocated than adding that same bandwidth to an existing machine, so the machine is essentially free).

So now we're swimming in CPUs, RAM, and SSD storage, and we're renting—no joke—the cheapest possible hardware we can at our provider (OVH). And no, we're not serving images, audio, or video—our CDN handles that (although see below for why I dropped S3). I actually do public-key authenticated encryption on every single packet and I've still got tons of CPU to spare.

Honestly, I feel a little bad about the whole thing. Why should I use 600 boxes when I really could do it in about 12? It just feels incredibly wasteful.

To partially make up for it, I put the 2x3TB of rusty metal storage we get with each of those tiny 600 boxes to good use, by moving us off S3 (which'll save another $100K/year, and a lot more as we continue to grow), so it's not been a total waste. Maybe I should look into Bitcoin mining next? :)

> But your average slow code wont perform well on 10Gb.

Yup, that really does require looking at things differently, measuring things differently. You actually need to have some kind of mechanical sympathy to remain efficient. Or build off of a project that does that for you, like Snabb Switch[0].

[0] https://github.com/SnabbCo/snabbswitch

Re: The C10M problem

#93

Post by barrkel »

alberth wrote: post #87

WhatsApp is achieving ~3M concurrent connections on a single node. [1][2] The architecture is FreeBSD and Erlang. It does make me wonder, and I've asked this question before [3], why can WhatsApp handle so much load per node when Twitter struggled for so many years (e.g. Fail Whale)? [1] http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2... [2, slide 16] http://www.erlang-factory.com/upload/presentations/55…

The problem of 1:1 messaging is slightly different to Twitter, which is more m:n. 1:1 messaging can be handled reasonably easily with a mailbox per user, and there is no shared state. Messaging with m:n has different optimal patterns depending on the relative ratios of m and n. Twitter has many users with millions of followers; if Twitter used a 1:1 mailbox approach like a chat app, these users would be whole countries worth of load on their own.

That's not to say that Twitter's scaling issues where wholly forgivable. They weren't fatal to the service, but I don't think they were necessary with good design from the start. High popularity is a good problem to have though.

Re: The C10M problem

#94

Post by erichocean »

mh- wrote: post #89
erichocean wrote: post #19

Earlier quoted context omitted.

If by "connection", you mean TCP, probably not. But that's not the only way to maintain connections, and it's certainly not the only reliable network protocol. My latest project keeps every "connection" open at all times, but it's a custom UDP based reliable messaging protocol, not TCP. At Facebook's scale, we'd have the equivalent of one billion connections "open". It's easy to keep them open, despite changing IP ad…

Can I ask some questions about that? Extremely interested.. I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.) I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc. Are you only using UDP from devic…

Good questions.

I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything.

It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go clientserver.

All I've seen so far is the usual UDP stuff: dropped packets, re-ordered packets, and duplicate packets. Nothing out of the ordinary. Our network protocol handles those things without any difficulties.

We did it specifically because all of our clients are mobile devices, and we didn't want to have to do the lengthy TCP connection setup (or worse, SSL setup) each time the network changed—which is often.

The biggest downside of UDP at the moment is that Apple only allows TCP connections in the background. That seems like a silly decision, but whatever, it's what they've done. I may, someday, set up a bunch of TCP forwarders for iOS devices running in the background. Our messages can be decoded just fine over byte-oriented streams, so it wouldn't change much.

It's a tough call. On the one hand, our UDP clients do not need to reconnect, since the connection is always set up. So when they wake, they send a packet to the server (Hello), and the server immediately sends back any immediate updates, such as new chat messages.

Our read path is perhaps over-optimized, so it's exactly the network latency of one round trip to get the updates since you last opened the app. It takes longer to get the UI up in some cases, so that's why we haven't done the TCP background thing. For others, that might be a much more important consideration.

Re: The C10M problem

#95

Post by Aloisius »

What's the current state of internet switches? Back when I used to run the Napster backend, one of our biggest problems was that switches, regardless of whether or not they claimed "line-speed" networking, would blow up once you pumped too many pps at them. We went through every single piece of equipment Cisco sold (all the way to having two fully loaded 12K BFRs) and still had issues.

Mind you, this was partially because of the specifics of our system - a couple million logged in users with tens of thousands of users logging in every second pushing large file lists, a widely used chat system which meant lots of tiny packets, a very large number of searches (small packets coming in, small to large going out) and a huge number of users that were on dialup fragmenting packets to heck (tiny MTUs!).

I imagine a lot of the kind of systems you'd want 10M simultaneous connections for would hit similar situations (games and chat for instance) though I'm not sure I'd want to (I can't imagine power knocking out the machine or an upgrade and having all 10 million users auto-reconnect at once).

Re: The C10M problem

#96

Post by julespitt »

rwallace wrote: post #77
wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

As someone who learned to program on a machine with five kilobytes of RAM, and later had to put a fair bit of effort into unlearning some of the resulting habits: Yes, that kind of efficiency (in terms of machine resources) and inefficiency (in terms of human resources) is effectively obsolete for general-purpose programming . What of that? If your heart's desire is to work in a context where machine resources are sc…

VIC-20 syndrome - I likely have it.

Re: The C10M problem

#97

Post by chubot »

lukego wrote: post #79
nl wrote: post #52

Earlier quoted context omitted.

Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

I'm the Snabb Switch originator. The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack. Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing I…

How about this use case: I have a ChromeCast on my home network, but I want sandbox/log its traffic. I would want to write some logic to ignore video data, because that's big. But I want to see the metadata and which servers it's talking to. I want to see when it's auto-updating itself with new binaries and record them.

Is that a good use case for Snabb Switch, or is there is an easier way to accomplish what I want?

Re: The C10M problem

#98

Post by nousernamesleft »

_delirium wrote: post #69
nl wrote: post #52

Earlier quoted context omitted.

Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

Another downside is that if you do networking in userspace, portability becomes your problem. If you do TCP/IP in the kernel, it works on nearly everything: just about any brand of Ethernet card, and even exotic non-Ethernet stuff (ISDN, whatever). If you do it in Snabb Switch, according to the wiki it currently supports exactly one class of interface: Intel-branded ethernet cards. Of course, I expect that will expan…

Realistically that isn't much of an issue. If you are doing something at this level, you pick your hardware for the task. You don't need to support every random old dell PC someone wants to use like a general purpose OS has too (linux, the BSDs, etc). And of course, intel cards are not hard to find.

Re: The C10M problem

#99

Post by chubot »

wpietri wrote: post #13

On the one hand, I love this. There's an old-school, down-to-the-metal, efficiency-is-everything angle that resonates deeply with me. On the other hand, I worry that just means I'm old. There are a lot of perfectly competent developers out there that have very little idea about the concerns that motivate thinking like this C10M manifesto. I sometimes wonder if my urge toward efficiency something like my grandmother's…

Having dealt with applications spanning thousands of machines, I'd say there will always be value to optimizing programs on a single machine. The prerequisite to distributed computing is being able to program a single computer.

If you can save an order of magnitude in machines, you will always avoid some scaling problem. 10 machines, 100 machines, and 1000 machines all have different problems.

Re: The C10M problem

#100

Post by PhasmaFelis »

logicchains wrote: post #39
PhasmaFelis wrote: post #33

Earlier quoted context omitted.

A beautifully carved chair may display excellent craftsmanship, but if you spend three months carving the world's most beautiful chair when the client asked for a dozen basic ladderbacks for a dinner party next Friday, you're not a very wise craftsman. Similarly, if your software runs in 1% of CPU on a typical customer's machine, spending 10x the time and resources to make it run in 0.1% is not laudable. Context is e…

>Similarly, if your software runs in 1% of CPU on a typical customer's machine, spending 10x the time and resources to make it run in 0.1% is not laudable. No, but spending 1.2x the time and resources might be. That's not an unreasonable proposition; something written in Go, C# or Scala can run 10 to 100 times faster than the equivalent code in Ruby, but it certainly doesn't take 10 to 100 times longer to write the c…

True! I'm just saying that good craftsmanship lies in understanding the situation and finding an appropriate balance. Ultra-efficient code is virtuous, but it's not the only virtue.
Post reply on HN