Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The C10M problem

c10m.robertgraham.com

Post reply on HN
Print view View original

101–110 of 116 posts

Re: The C10M problem

#101

Post by Scaevolus »

erichocean wrote: post #94
mh- wrote: post #89

Earlier quoted context omitted.

Can I ask some questions about that? Extremely interested.. I've considered doing something similar for our messaging/signalling protocol (currently standard TCP sockets established to several million mobile devices.) I had concerns about what the deliverability of UDP would be on mobile networks; many carriers are going towards NAT'ing everything, (forced) transparency proxies, etc. Are you only using UDP from devic…

Good questions. I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything. It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client ser…

A lot of these features sound similar to MinimaLT: http://cr.yp.to/tcpip/minimalt-20130522.pdf

Do you have explicit DoS protections?

Re: The C10M problem

#102

Post by erichocean »

Scaevolus wrote: post #101
erichocean wrote: post #94

Earlier quoted context omitted.

Good questions. I'm using UDP in both directions, and I do have a heartbeat (currently set at 30 seconds, but I think we could go to 60 seconds without any problems). We do use that to keep track of IP:PORT changes, but also (mainly?) to keep the UPD hole punched, due to carrier's NAT'ing everything. It works, and it works well. It's the same idea behind WebRTC, except instead of going peer-to-peer, you go client ser…

A lot of these features sound similar to MinimaLT: http://cr.yp.to/tcpip/minimalt-20130522.pdf Do you have explicit DoS protections?

That paper (as well as CurveCP) was definitely a huge inspiration for what I'm doing.

A big difference is I'm not running a packet scheduler (e.g. Chicago). Frankly, our data rate on a per device basis is just minuscule, and our internal protocol has back pressure built into it anyway, so I just skipped that part. If it becomes an issue (unlikely), I'll of course actually add an explicit scheduler so our UDP traffic plays nicely with others.

I'm not doing MLT's puzzle step (although I really like the concept). At the moment, all I can do is deny connections from unknown devices if we're under attack (I can drop packets from an unknown device with a single hash lookup). We're also in the process of moving to OVH, which is able to block DDoS stuff at the network edge, should it happen.

There's more stuff I've got planned, but that's it for now.

Re: The C10M problem

#103

Post by lukego »

chubot wrote: post #97
lukego wrote: post #79

Earlier quoted context omitted.

I'm the Snabb Switch originator. The project is new: I and other open source contributors are currently under contract to build a Network Functions Virtualization platform for Deutsche Telekom's TeraStream project [1] [2]. This is called Snabb NFV [3] and it's going to be totally open source and integrated with OpenStack. Currently we are doing a lot of virtualization work from the "outside" of the VM: implementing I…

How about this use case: I have a ChromeCast on my home network, but I want sandbox/log its traffic. I would want to write some logic to ignore video data, because that's big. But I want to see the metadata and which servers it's talking to. I want to see when it's auto-updating itself with new binaries and record them. Is that a good use case for Snabb Switch, or is there is an easier way to accomplish what I want?

That sounds pretty reasonable to me.

If you can express how you want to filter with a fancy pcap-filter expression the tcpdump is the easy answer. Otherwise you might want to code it up in Lua with snabbswitch.

Here is our basic trace store/replay library today: https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/p...

Re: The C10M problem

#104

Post by nkurz »

rdtsc wrote: post #30
erichocean wrote: post #24

Earlier quoted context omitted.

The entire purpose is to NOT route stuff through the kernel. TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores. Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor,…

Linux will often still schedule kernel threads to run on those cores so they are not totally isolated. Also cache effects. If your architecture shares caches between cores, sometimes it would be worth "wasting" a neighboring core to avoid ssh and gdb thrashing your cache. Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle. There is an interesting research done by Siemens, that takes…

Oh and also don't forget to set up IRQ affinity to avoid any of those cores to handle.

Perhaps this is what you meant, but this is straightforward. Simply disabling 'irqbalance' is a simple way to do this. Alternatively, you can also configure it to cooperate with 'isolcpus' by using 'FOLLOW_ISOLCPUS'.

Re: The C10M problem

#105

Post by srean »

sitkack wrote: post #90
srean wrote: post #42

Earlier quoted context omitted.

I think the "just good enough for business" attitude contributed to the demise of the American car and the rise of the Japanese ones. The American tradition was to use engineering tolerances that would maximize throughput under the constraint that it produced a pretty functional car. The Japanese tradition on the other hand was to use tighter tolerances, well because there was room for tightening the tolerance. At th…

I think you are conflating the demise of the American car, "just good enough for business" and the fetishization of over tolerancing. All of those things are much more complex than "a tradition" of quality. I would assert that Japanese quality is a rather new thing and that it ebbs and flows with the time and the needs. By your measure, the Zero was a low quality plane, but served the MBAs in the Japanese military we…

As far as AK-47 goes I think you are conflating two different engineering concepts, perhaps because you are not an engineer. One concept is that of "clearance" as a feature for the designed functionality of a product vs loose tolerance as a result of sloppiness.

I think, you think tolerance means fits that rattle, thats not what it means in engineering. Loose tolerance may just as likely lead to interference fits.

Forget AK-47, there are more egregious examples, consider the Blackbird, SR-71 one of the fastest military aircrafts to have graced the skies, a marvel of engineering: it had just wide gaps between the parts of its metallic skin, at the joints. Now going by your logic one might think SR71 became successful because of loose tolerance and "good enough". It was quite the opposite, the clearances were deliberate and necessary for its success. They were there to account for thermal expansion. For the AK it was necessary for rough use and low to zero maintenance.

In fact not having clearance for SR71 or the AK would have constituted intellectual sloppiness: going with cookie cutter decisions and not optimizing the product for the use case. What looks sloppy to an amateur might actually be the result of perfection by an expert.

May I recommend Zen and the Art of Motorcycle Maintenance to you.

Further, I think you are being obtuse and defensive and railing against something I have not said. I am not quite sure why. I have never claimed all products need to be finished to the point of "burnished wood knobs". My commentary was on personal growth as an engineer. If it doesn't bother you somewhere to turn in a product that you know you could improve with little effort, you are not going to be a quality engineer, and will not be able to produce a quality product when one such is desired. I am categorically not saying that each item that you deliver has to be the epitome of some arbitrary quality standard.

The cultural tradition that I was talking about was not about adding cost to the product by unnecessary finish. Often people do not finish the product even when it would not have taken much effort. This is rationalized with the logic that the finish would have little immediate value, because it is already good enough for the job, but far from "good". The other reason is sometimes the craftsman just does not have the skill and "good enough" is a good argument to take cover under.

Re: The C10M problem

#106

Post by wmf »

Aloisius wrote: post #95

What's the current state of internet switches? Back when I used to run the Napster backend, one of our biggest problems was that switches, regardless of whether or not they claimed "line-speed" networking, would blow up once you pumped too many pps at them. We went through every single piece of equipment Cisco sold (all the way to having two fully loaded 12K BFRs) and still had issues. Mind you, this was partially be…

10 Gbps switches are pretty good and are generally line rate (as long as you avoid ten-year-old chassis).

Re: The C10M problem

#107

Post by chubot »

lukego wrote: post #103
chubot wrote: post #97

Earlier quoted context omitted.

How about this use case: I have a ChromeCast on my home network, but I want sandbox/log its traffic. I would want to write some logic to ignore video data, because that's big. But I want to see the metadata and which servers it's talking to. I want to see when it's auto-updating itself with new binaries and record them. Is that a good use case for Snabb Switch, or is there is an easier way to accomplish what I want?

That sounds pretty reasonable to me. If you can express how you want to filter with a fancy pcap-filter expression the tcpdump is the easy answer. Otherwise you might want to code it up in Lua with snabbswitch. Here is our basic trace store/replay library today: https://github.com/SnabbCo/snabbswitch/blob/master/src/lib/p...

OK and I forgot to say I might want to deny some traffic... like disable auto updates but still allow it to contact other servers to play video. AFAIK tcpdump doesn't let you do that.

Thanks for the very cool project! I will have to learn more about it.

Re: The C10M problem

#108

Post by nkurz »

erichocean wrote: post #92
justincormack wrote: post #65

Earlier quoted context omitted.

I think there has been a renewed interest around 10Gb ethernet (and SSD). Basically computers were grossly limited by IO with gigabit ethernet and hard drives, you could buy a pretty low end box and saturate them, instead people bought high end boxes and ran slow code. But your average slow code wont perform well on 10Gb.

Indeed. I've got an architecture right now that I could actually do everything I need to do in a single rack with 40GigE adaptors, except...I can't (easily) get that kind of bandwidth to so few boxes using third parties, and more importantly, I can't get the bandwidth cheaply that way . So instead, I'm scaling out with mostly idle Intel E5-1620v2 4 core (8HT) boxes and single 1GbE connections because that's about 70%…

To partially make up for it, I put the 2x3TB of rusty metal storage we get with each of those tiny 600 boxes to good use, by moving us off S3 (which'll save another $100K/year, and a lot more as we continue to grow), so it's not been a total waste. Maybe I should look into Bitcoin mining next? :)

Not necessary if you can displace your own purchases, but my first thought was that maybe you could do something like "Amazon Glacier", offering distributed but rarely used storage for a fee. I presume that Amazon offers this because they have a similar surplus of live empty disk space.

I know you are joking about the Bitcoin mining, but my second thought was that maybe there is a parallel 'proof of storage' idea. There could be a public market for backup storage, where instead of being paid for 'proof of work' one is randomly tested for 'proof of backup'. You say you'll store something, and you are paid based on your ability to answer random 'challenge' requests in a timely manner.

And then I noticed another front page article on HN from someone writing software that could do something very similar: http://hypered.io/blog/2014-02-17-building-reesd/ Perhaps you could be his backend.

Re: The C10M problem

#109

Post by sitkack »

srean wrote: post #105
sitkack wrote: post #90

Earlier quoted context omitted.

I think you are conflating the demise of the American car, "just good enough for business" and the fetishization of over tolerancing. All of those things are much more complex than "a tradition" of quality. I would assert that Japanese quality is a rather new thing and that it ebbs and flows with the time and the needs. By your measure, the Zero was a low quality plane, but served the MBAs in the Japanese military we…

As far as AK-47 goes I think you are conflating two different engineering concepts, perhaps because you are not an engineer. One concept is that of "clearance" as a feature for the designed functionality of a product vs loose tolerance as a result of sloppiness. I think, you think tolerance means fits that rattle, thats not what it means in engineering. Loose tolerance may just as likely lead to interference fits. Fo…

You keep using that word. I do not think it means what you think it means.

Re: The C10M problem

#110

Post by chubot »

erichocean wrote: post #24
stefan_kendall wrote: post #22

Earlier quoted context omitted.

Not really. Last I checked, stock Ubuntu was choking around 60k concurrent connections for no reason, and Fedora could handle a lot more. This was a couple years back, but I'd demand numbers before assuming the situation has changed.

The entire purpose is to NOT route stuff through the kernel. TFA explains how to do it, and I've done it myself. You can set a flag on the Linux kernel when it boots limiting it to the first N cores. I usually use 2. The remaining cores are completely idle—Linux will not schedule any threads on those cores. Then you build an app that works more-or-less like Snabb Switch, which talks directly to the Ethernet adaptor,…

This sounds really cool, but I didn't see where the original post talks about this hybrid setup with the Linux kernel (reading the first part and skimming through the rest, which I had already read back when it came out).

Do you know of other resources for this approach? I have seen cpusets but not used them.

Post reply on HN