Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happens if you write a TCP stack in Python?

jvns.ca

Post reply on HN
Print view View original

41–50 of 125 posts

Re: What happens if you write a TCP stack in Python?

#41

Post by collyw »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

I was surprised by this as well. The common quote is that Python is 10 - 100 times slower than C, yet computers have been doing TCP for ages, and Moores law has meant that my phone is probably more powerful / faster than my laptop from ten years ago. It didn't quite add up.

Re: What happens if you write a TCP stack in Python?

#42

Post by marcosdumay »

kyllo wrote: post #28
lmm wrote: post #7

Earlier quoted context omitted.

The answer to questions about performance is always "profile it". I mean, sure, you could probably get a crude speedup by just running it under pypy or some such, but the real answer is to profile it, see where the bottleneck is, and fix it. As the other post suggests, this is not a problem in all high-level languages; it's possible to write very high-performance code in Haskell or Ocaml, for example. But python's se…

So you're saying that late binding is the most significant reason why Python is slow? Is slowness just an inherent tradeoff in using a language that supports this powerful feature?

Not only late biding, but all the Python's powerfull reflection (not only hashes, but it's string hashes all the way down) makes it hard to write a fast compiler.

Not impossible, as lmm said, just hard.

Re: What happens if you write a TCP stack in Python?

#43

Post by easytiger »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

scapy is the slowest thing on earth. It shouldn't be used.

Re: What happens if you write a TCP stack in Python?

#44

Post by throwaway0010 »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

Indeed. I can negotiate a connection to google.com:80 successfully on a 1000ms latency pipe just fine. Slowly, but fine.

Re: What happens if you write a TCP stack in Python?

#45

Post by crazypyro »

jnazario wrote: post #21
crazypyro wrote: post #3

Earlier quoted context omitted.

Implement it in C. Also its likely more a python problem than a "high level" language problem.

there's a userland stack written in C in honeyd. http://honeyd.org/ (slow to respond) and http://en.wikipedia.org/wiki/Honeyd

Except that was written for an explicit purpose that wasn't speed. I fail to see how my suggestion of implementing low level networking in C and linking that to python code is considered a poor solution.

Re: What happens if you write a TCP stack in Python?

#46

Post by malone »

I like your solution to prevent the kernel from interfering with your packets.

An alternative method I've used in the past is to add an iptables rule to silently drop the incoming packets. libpcap sees the packets before they are dropped, so you'll still be able to react to them, but the kernel will ignore them (and therefore won't attempt to RST them).

Re: What happens if you write a TCP stack in Python?

#47

Post by bane »

tptacek wrote: post #23

The idea that Python is so slow that it's confusing TCP sounds wrong to me. I think it's more likely that your packet capture scheme is slow. It looks like you're using scapy, which I assume is in turn using libpcap... which may be buffering (in high-performance network monitoring, the packet capture interface goes out of its way to buffer). Which is something you can turn off. About 13 years ago, I wrote my own prog…

If a microcontroller and a Commodore 64 can do TCP/IP, Python on a modern PC can handle it.

https://en.wikipedia.org/wiki/UIP_(micro_IP)

http://www.techrepublic.com/blog/classics-rock/surf-the-web-...

Re: What happens if you write a TCP stack in Python?

#48

Post by crazypyro »

mkonecny wrote: post #39
crazypyro wrote: post #3

Earlier quoted context omitted.

Implement it in C. Also its likely more a python problem than a "high level" language problem.

This has nothing to do with Python or C. More likely he has a bug in his code.

Python being magnitudes slower than C for networking code has a lot to do with Python, actually. Other smarter people have already explained it better than I can up above.

Re: What happens if you write a TCP stack in Python?

#50

Post by milesvp »

kyllo wrote: post #28
lmm wrote: post #7

Earlier quoted context omitted.

The answer to questions about performance is always "profile it". I mean, sure, you could probably get a crude speedup by just running it under pypy or some such, but the real answer is to profile it, see where the bottleneck is, and fix it. As the other post suggests, this is not a problem in all high-level languages; it's possible to write very high-performance code in Haskell or Ocaml, for example. But python's se…

So you're saying that late binding is the most significant reason why Python is slow? Is slowness just an inherent tradeoff in using a language that supports this powerful feature?

There's probably something to this statement. there was a great article I saw on HN about a year ago that was talking about this (wish I'd bookmarked it). Crux of it was he looked at idiomatic python, saw all the hash lookups that entailed and said if you wrote C like that, it'd be slow as hell too. he then proceeded to speed up an algorithm to near C speeds by removing structures relying on these lookups. Was quit amazing to see.

Keep in mind that in current computer architectures memory requests are very slow. And any data structure that randomizes memory access means that you have a good chance of a cache miss and now have to hit even slower ram.

Post reply on HN