Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if you write a TCP stack in Python?

jvns.ca

Post reply on HN
Print view View original

61–70 of 125 posts

Re: What happens if you write a TCP stack in Python?

#61

Post by peterwwillis »

You don't need to spoof a different MAC or IP to implement your own stack on a raw socket, Python is not too slow to handle negotiating a connection, and your interpretation of how tcp/ip works is flawed. I highly recommend you read a good book about tcp/ip and learn how the kernel works with network applications of different types.

In terms of using Scapy for your packet crafting, here are some guides with examples that may help you work around your issues. (Hint: use the Scapy-defined sending and receiving routines and don't implement your own, or stop using Scapy and implement your own raw packet sockets) http://securitynik.blogspot.com/2014/05/building-your-own-tc... https://github.com/yomimono/client-fuzzball/blob/master/fuzz... https://www.sans.org/reading-room/whitepapers/detection/ip-f... http://www.lopisec.com/2011/12/learning-scapy-syn-stealth-po...

Re: What happens if you write a TCP stack in Python?

#62

Post by kashif »

kashif wrote: post #52

I have actually implemented a TCP stack in python, unfortunately I can't share it publicly just yet. The authors problems are because she is not using RAW_IP_SOCKETS. Making TCP packets using pythons struct module is a breeze. I can post specific examples in code if anyone is interested. Finally you can write a proper TCP stack in python, there is no reason not to. Your user-space interpreted stack won't be as fast a…

Google sends extra packets because it does not comply with TCP Slow Start(RFC 2581) - if the author implements a proper tcp state machine and buffer then she would be able to handle this.

Re: What happens if you write a TCP stack in Python?

#63

Post by lnkmails »

As someone who worked on writing protocol specs as code for simulation purposes, I can see how much fun this is. A pure python network simulator (ns2 is C++ and painfully hard to debug) would actually be nice and encourage a lot of theoreticians to get into real programming. I've spent a reasonable amount of time in the industry building distributed systems and I can say with confidence understanding low level protocols improves your thinking.

Re: What happens if you write a TCP stack in Python?

#64

Post by beneater »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

This seems likely, but also easy to work around by advertising a smaller window size.

When you send ACKs, not only do you send the acknowledgement number indicating which byte you expect next, but you also send a window size indicating how many bytes you're willing to receive before the remote end has to wait for another acknowledgement. Normally you want this to be somewhat large so you don't spend lots of idle time waiting around for ACKs. (But not so large that packets get dropped). This is the key to TCP flow control, which was kinda glossed over in the blog post in the interest of keeping things simple.

But perhaps by default, you're advertising a too-large window considering the circumstances. I bet you could make this a lot more reliable just by advertising something smaller.

Good TCP implementations have overcome a lot more than some unwanted buffering.

Re: What happens if you write a TCP stack in Python?

#66

Post by tptacek »

beneater wrote: post #64
tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

This seems likely, but also easy to work around by advertising a smaller window size. When you send ACKs, not only do you send the acknowledgement number indicating which byte you expect next, but you also send a window size indicating how many bytes you're willing to receive before the remote end has to wait for another acknowledgement. Normally you want this to be somewhat large so you don't spend lots of idle time…

If it's the buffering delay I'm thinking of, and your packet capture filter is tight, the delay we're talking about can be multiple seconds long. Changing window sizes won't fix it.

Re: What happens if you write a TCP stack in Python?

#67

Post by Dylan16807 »

sly010 wrote: post #33

If an 8MHz microcontroller is fast enough to implement TCP, then Python should be fast enough too. Here is my two cents on the expirement: 1. You don't really have to ack every packet, you have to order them, drop duplicates and ack the last one. 2. Google ignores the TCP flow control algorithm and sends the first few hundred packets very quickly without waiting for acks. They do this to beat the latency of the flow…

Even 8MHz is massive overkill. It's almost fast enough to bit-bang ethernet.

Re: What happens if you write a TCP stack in Python?

#68

Post by mcguire »

milesvp wrote: post #50
kyllo wrote: post #28

Earlier quoted context omitted.

So you're saying that late binding is the most significant reason why Python is slow? Is slowness just an inherent tradeoff in using a language that supports this powerful feature?

There's probably something to this statement. there was a great article I saw on HN about a year ago that was talking about this (wish I'd bookmarked it). Crux of it was he looked at idiomatic python, saw all the hash lookups that entailed and said if you wrote C like that, it'd be slow as hell too. he then proceeded to speed up an algorithm to near C speeds by removing structures relying on these lookups. Was quit a…

Practically every operation in Python starts with a hashtable lookup. Global and local environments are hashtables, modules are hashtables, objects are hashtables. Eventually you get down to C, which is usually very good---the hashtable implementation in particular---but for a lot of Python code the extra indirections are expensive. That's also why it is hard to optimize.

In this particular case, I don't think it's the problem, though. Just something to keep in mind.

Re: What happens if you write a TCP stack in Python?

#69

Post by danudey »

tptacek wrote: post #32
jvns wrote: post #29

Earlier quoted context omitted.

my favorite thing about writing blog posts is comments like this. Thank you! I didn't consider that the packet capture interface might do buffering. That might explain a lot of the problems I was having :)

This is a great project. Keep playing with it! You might find that the serverside of TCP is more useful to have in Python than the clientside (having a userland IP/TCP serverside allows you to create fake hosts out of thin air).

This sounds super handy for situations like honeypots. Take a /24, assign the hosts you actually have, and then spoof the rest to another server which fakes open SMTP, HTTPS, etc. connections, or which replies to every connection attempt (slowly!) with a successful (if slow!) open.

There's an iptables module called tarpit[1], which takes advantage of some peculiarities of the TCP protocol to essentially prevent the remote host from closing the connection, which can force (conforming?) TCP clients to take 12-24 minutes to timeout every connection. It can make portscanning unpleasantly expensive and time-consuming.

[1] http://linux.die.net/man/8/iptables

Re: What happens if you write a TCP stack in Python?

#70

Post by beagle3 »

While not immediately relevant, if you find this discussion interesting, have a look at sshuttle:

sshuttle[0] is a pure-python one-way TCP VPN solution that is very well behaved and quite efficient. The source is highly readable as well. +1 to everything Avery Pennarun has released (including wvdial, bup)

[0] https://github.com/apenwarr/sshuttle

Post reply on HN