Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if you write a TCP stack in Python?

jvns.ca

Post reply on HN
Print view View original

111–120 of 125 posts

Re: What happens if you write a TCP stack in Python?

#111

Post by codygman »

lutusp wrote: post #104
rectangletangle wrote: post #103

Earlier quoted context omitted.

Python may be slow, but it's only relatively slow. It's still fast enough for the vast majority of use cases.

As with most things in software, speed can't be measured only by the running program -- the time required to write the program should be included in nearly all cases. Python: fast to write, slow to run. C/C++: slow to write, fast to run. To put it succinctly, you can write fast programs, and you can write programs fast, but you can't write fast programs fast.

> To put it succinctly, you can write fast programs, and you can write programs fast, but you can't write fast programs fast.

Haskell, Clojure, and Ocaml do pretty well at writing fast programs fast for what I consider appealing values of fast.

Re: What happens if you write a TCP stack in Python?

#112

Post by donavanm »

srean wrote: post #91

Too late to join the story, but I am really curious if datacenter nodes intended for heavy mapreduce use implement this layer in user space. The bottleneck for such processes is typically network I/O and I can imagine that taking control of the network in the user space might offer some modest to significant wins. For Hadoop in particular network packets needs to traverse quite a few layers before it is accessible to…

There are a couple implementations of userland ip/tcp out there. As I recall at least one or two are fairly direct ports of a BSD network stack in to a library. These pair up with concepts like intels dpdk to move networking closer to the business logic.

Here are some slides for one version: http://www.bsdcan.org/2014/schedule/attachments/260_libuinet...

Edit: with regards to motivation it's always been something along the lines of a network appliance that I've seen. The mainline linux network stack is more than capable of doing many millions of packets per second over hundreds of thousands of concurrent streams. The network stack will not be the limitation in something like batch processing.

Re: What happens if you write a TCP stack in Python?

#114

Post by zerop »

milkshakes wrote: post #73
danudey wrote: post #69

Earlier quoted context omitted.

This sounds super handy for situations like honeypots. Take a /24, assign the hosts you actually have, and then spoof the rest to another server which fakes open SMTP, HTTPS, etc. connections, or which replies to every connection attempt (slowly!) with a successful (if slow!) open. There's an iptables module called tarpit[1], which takes advantage of some peculiarities of the TCP protocol to essentially prevent the r…

this is in fact exactly how honeyd works: http://www.honeyd.org/

Thanks for mentioning Honeyd. Are there any more software like this.

Re: What happens if you write a TCP stack in Python?

#115

Post by digitalsushi »

qewrffewqwfqew wrote: post #99
digitalsushi wrote: post #79

Earlier quoted context omitted.

Our tcp stack is written in tcl. The window size is set to 1. It works fine.

As a heavy user of Tcl, I'm interested to know more. Is this related to hping3 or pktsrc or NS? Or are you F5?

It's pktsrc.

Re: What happens if you write a TCP stack in Python?

#116

Post by VLM »

phaker wrote: post #85
Dylan16807 wrote: post #67

Earlier quoted context omitted.

Even 8MHz is massive overkill. It's almost fast enough to bit-bang ethernet.

Some guy actually did it on a 20MHz avr: http://www.cesko.host.sk/IgorPlugUDP/IgorPlug-UDP%20%28AVR%2... (hope the url doesn't get borked). Sadly 10BASE-T ethernet uses manchester coding, so you need 20MHz :( you could pull it off with a slower processor if you had a fifo that can reach 20Mbaud on the serial end, but the usart in avr micros runs at a fraction of the system clock.

"you could pull it off with a slower processor if you had a fifo that can reach 20Mbaud on the serial end"

For a surprise, google for / research how wiznet based ethernet controllers are used on Arduino shields or just wired up by hand (I did a hand wired wiznet 5100 once upon a time...). Yes yes its possible to run it in full TCP or UDP termination mode which is the way most people use it, but it can also terminate at IP level, and with some limitations, raw ethernet packet level. If you pull that data sheet and want to actually try this, look for documentation about "MACRAW" mode. This basically turns the controller into an ethernet packet FIFO as you describe with ethernet on one side and SPI bit banging on the other side.

Disclaimer: I've never done anything with MACRAW mode on wiznet controllers other than read about it in the data sheet and wonder why I'd want to do that. If I was going to blackhat a weird portable appliance or otherwise do heavy ethernet weirdness I'd use a rasp pi or similar SBC and linux not arduino compiled C, but whatever.

Re: What happens if you write a TCP stack in Python?

#117

Post by e12e »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

Python being too slow for tcp/ip sounds silly. Afaik the c64 can run a webserver. At what... 4mhz? Less? I think you could probably emulate the c64 tcp/ip stack in python in realtime+ (or -...) on a multighz cpu...

Re: What happens if you write a TCP stack in Python?

#118

Post by MagerValp »

e12e wrote: post #117
tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

Python being too slow for tcp/ip sounds silly. Afaik the c64 can run a webserver. At what... 4mhz? Less? I think you could probably emulate the c64 tcp/ip stack in python in realtime+ (or -...) on a multighz cpu...

1 MHz, and while I only implemented UDP in my stack (https://github.com/oliverschmidt/ip65), Jonno Downes took over and added TCP later. There's also Adam Dunkels uIP and Doc Bacardi's HTTP-Load.

So yes, if a C64 can handle it Python should have plenty of power.

Re: What happens if you write a TCP stack in Python?

#119

Post by MagerValp »

tptacek wrote: post #59
blutoot wrote: post #55

Earlier quoted context omitted.

Your "userland TCP/IP" comment got me excited and I started digging around. I wonder if there's some benefit in trying to port something like TCP Daytona [1,2] in a high-level language like Python. I'm curious if that will somehow advance the adoption of SDNs. [1] https://github.com/jamesbw/tcp-daytona [2] http://nms.lcs.mit.edu/~kandula/data/daytona.pdf

Like writing your own emulator, writing a full TCP stack is a project that is intrinsically worth doing. You can probably also come up with real-world applications for it (as a security tester, there are lots of applications for having full control over a TPC stack, regardless of how performant it is), but just having done it offers a huge learning return on a modest investment. You probably don't fully grok what TCP…

I just want to second this, I didn't fully understand networking until I wrote my own stack and threw packets on the wire. A simple stack with ARP/ICMP/IP/UDP can be written in a day and seeing your very own ping packets fly across the globe is pretty damn awesome.

Re: What happens if you write a TCP stack in Python?

#120

Post by tjz77 »

tptacek wrote: post #66
beneater wrote: post #64

Earlier quoted context omitted.

This seems likely, but also easy to work around by advertising a smaller window size. When you send ACKs, not only do you send the acknowledgement number indicating which byte you expect next, but you also send a window size indicating how many bytes you're willing to receive before the remote end has to wait for another acknowledgement. Normally you want this to be somewhat large so you don't spend lots of idle time…

If it's the buffering delay I'm thinking of, and your packet capture filter is tight, the delay we're talking about can be multiple seconds long . Changing window sizes won't fix it.

I haven't read your code yet but I was curious about which packet you are ACK'ing?

Is your ACK sequence number the sum of all received data lengths? I think that is how that works?

Post reply on HN