Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The C10M problem

c10m.robertgraham.com

Post reply on HN
Print view View original

81–90 of 116 posts

Re: The C10M problem

#81

Post by lukego »

_delirium wrote: post #69
nl wrote: post #52

Earlier quoted context omitted.

Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

Another downside is that if you do networking in userspace, portability becomes your problem. If you do TCP/IP in the kernel, it works on nearly everything: just about any brand of Ethernet card, and even exotic non-Ethernet stuff (ISDN, whatever). If you do it in Snabb Switch, according to the wiki it currently supports exactly one class of interface: Intel-branded ethernet cards. Of course, I expect that will expan…

True. Or we might end up sharing drivers between projects too.

In my experience high-end systems choose hardware to match software, so you only need to support one suitable option. The main reason we would add support for a new NIC is if it turns out to be better in some way e.g. as Mellanox support 40G now and Intel don't.

In Snabb Switch have a lot of 10G/40G NICs online for people to play with now: https://groups.google.com/d/msg/snabb-devel/PXOsv0uLQCE/HjPj...

Re: The C10M problem

#82

Post by cjbprime »

> There is no way for the primary service (such as a web server) to get priority on the system, leaving everything else (like the SSH console) as a secondary priority.

Just for the record -- the SSH console is the primary priority. If the web server always beats the SSH console and the web server is currently chewing 100% CPU due to a coding bug..

Re: The C10M problem

#83

Post by MrBuddyCasino »

lukego wrote: post #79
nl wrote: post #52

Earlier quoted context omitted.

Snabb Switch looks awesome. Is there a list of "things" (not sure of the terminology?) people have built with it? I guess the downside is that you can't virtualize it (I realize that is kind of the point, but it does reduce the accessibility of it).

I'm the Snabb Switch originator. The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack. Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing I…

I'm clueless, so I'll just ask: is this made for software-defined networking?

Re: The C10M problem

#84

Post by lukego »

MrBuddyCasino wrote: post #83
lukego wrote: post #79

Earlier quoted context omitted.

I'm the Snabb Switch originator. The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack. Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing I…

I'm clueless, so I'll just ask: is this made for software-defined networking?

No. Honestly, software defined networking is the idea of replacing all of your networking staff with a very clever distributed Java program. Half the world thinks this is genius and the other half thinks it's a facepalm.

Network Functions Virtualization is the idea of replacing networking boxes (Cisco, Juniper, Ericsson, F5, ...) with virtual machines running on your own PCs. This is basically a "private cloud" but with emphasis on doing networking in a way that doesn't annoy ISPs.

Rest assured that Software Defined Networking will be redefined to mean whatever the next good technology turns out to be :-)

Re: The C10M problem

#85

Post by eranation »

What about academic operating system research that was done years ago? Exokernel, SPIN, all aim to solve the "os is the problem" issue. Why don't we see more in that direction?

Re: The C10M problem

#86

Post by Shorel »

dbenhur wrote: post #43
electrograv wrote: post #25

Earlier quoted context omitted.

It really just depends on how much of a computational surplus you have. Some tasks have become so incredibly easy relative to today's computational horsepower, that we can afford waste 90% of it if it means we can be 10% more productive. But this isn't true everywhere. Imagine if Google's servers cost 100x more to run, if they didn't spend time to make their code efficient. Imagine if your video games ran at a frame…

> when languages like Python and Ruby can be over 100x slower than other high level languages You said a wrong thing. Java is a far cry from a high-level language. Javascript is arguably a high-level language however, and meaningless benchmarks (aka "damn lies") attest V8 is in spitting distance of Java. http://benchmarksgame.alioth.debian.org/u32/javascript.php Scala is much higher level than Java, compiles to the s…

We can revive the old 'language generations' concept I was taught in school:

- First generation: binary opcodes entered by hand.

- Second generation: Assembler languages.

- Third generation: Compiled languages like C++.

- Fourth generation: Interpreted high level like Python and Ruby. Also SQL.

- Fifth generation: There's no such thing. Or may be Apple's Siri is.

Java seems to be in a "3.5" generation, if that thing can be conceived. Javascript started as fourth and keeps getting a little bit closer to third.

Re: The C10M problem

#87

Post by alberth »

WhatsApp is achieving ~3M concurrent connections on a single node. [1][2]

The architecture is FreeBSD and Erlang.

It does make me wonder, and I've asked this question before [3], why can WhatsApp handle so much load per node when Twitter struggled for so many years (e.g. Fail Whale)?

[1] http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2...

[2, slide 16] http://www.erlang-factory.com/upload/presentations/558/efsf2...

[3] https://news.ycombinator.com/item?id=7171613

Re: The C10M problem

#88

Post by sitkack »

guard-of-terra wrote: post #72
logicchains wrote: post #64

Earlier quoted context omitted.

Maybe German manufacturing would be a better example?

German quality sucked some half century earlier, so much that England had to require them write "Made in" on their goods to warn consumers. Quality comes and goes.

Exactly. When people complain about the quality of Chinese goods as a proxy for China and Chinese people as a whole, I ask them where their MacBook Pro is made and they quickly shut up.

All of these lazy, low tolerance sweeping generalizations need to go!

Re: The C10M problem

#89

Post by mh- »

erichocean wrote: post #19
chongli wrote: post #15

Earlier quoted context omitted.

If you could do 10M connections on one machine, then why not 1B on 100? Does it even make sense to have a billion simultaneous connections?

If by "connection", you mean TCP, probably not. But that's not the only way to maintain connections, and it's certainly not the only reliable network protocol. My latest project keeps every "connection" open at all times, but it's a custom UDP based reliable messaging protocol, not TCP. At Facebook's scale, we'd have the equivalent of one billion connections "open". It's easy to keep them open, despite changing IP ad…

Can I ask some questions about that? Extremely interested..

I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.)

I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc.

Are you only using UDP from device->infrastructure? If not, do you rely on the devices providing an ip:port over a heartbeat of sorts (to keep up with IP changes?)

Do you have any issues with deliverability, in either direction? (not due to UDP's inherent properties, but because of carrier network behavior)

thanks very much for anything you're able to answer.

Re: The C10M problem

#90

Post by sitkack »

srean wrote: post #42
usea wrote: post #34

Earlier quoted context omitted.

It is wasteful to spend time doing something that benefits nobody, or where the benefits outweigh the cost. Your time is worth something.

I think the "just good enough for business" attitude contributed to the demise of the American car and the rise of the Japanese ones. The American tradition was to use engineering tolerances that would maximize throughput under the constraint that it produced a pretty functional car. The Japanese tradition on the other hand was to use tighter tolerances, well because there was room for tightening the tolerance. At th…

I think you are conflating the demise of the American car, "just good enough for business" and the fetishization of over tolerancing. All of those things are much more complex than "a tradition" of quality. I would assert that Japanese quality is a rather new thing and that it ebbs and flows with the time and the needs.

By your measure, the Zero was a low quality plane, but served the MBAs in the Japanese military well for meeting its objectives. The AK47, one of the most successful machines in the world has extremely sloppy tolerances and it is exactly those lenient tolerances that enabled its success.

We always have to be cognizant of economics, over building, polishing or engineering a system is a waste. Calling it a craft doesn't make it any more acceptable. Proing-up and being good at what you do doesn't mean you need to put burnished wood knobs on your software.

Post reply on HN