Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Framework for faster distributed data transfer

github.com

Post reply on HN
Print view View original

1–10 of 12 posts

Re: Show HN: Framework for faster distributed data transfer

#3

Post by andrewcooke »

this is all new to me and i'm struggling to find a friendly, simple introduction to the field. does anyone have a good link to, say, an introductory article on fountain codes? i'm starting to get a feel for the idea from reading various wikipedia pages, paper abstracts, powerpoints, etc, but a good "layman's" article would be great...

[update] huh. just after posting i started searching for "fountain codes" and found this - http://blog.notdot.net/2012/01/Damn-Cool-Algorithms-Fountain... (previously had been searching for luby, raptor, etc).

Re: Show HN: Framework for faster distributed data transfer

#4

Post by kenko »

Change test-client.py to end:

  print client.fetch(1000)
  print client.fetch(1)
And when you run it, you get the same thing twice (whatever file was specified as the first argument to test-server.py on the commandline). On the other hand, change it to end

  print client.fetch(1)
  print client.fetch(1000)
and you get "Hello World!" twice.

At one point the server process also gave me this:

  Server(s) started. Press enter to stop.
  Exception in thread Thread-1:
  Traceback (most recent call last):
    File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
      self.run()
    File "/Users/wolfson/src/others/pouring-rain/LT/Server.py", line 84, in run
      self.streams[ entry[1] ].discard( entry[0] )
  NameError: global name 'entry' is not defined

Re: Show HN: Framework for faster distributed data transfer

#5

Post by mgronhol »

kenko wrote: post #4

Change test-client.py to end: print client.fetch(1000) print client.fetch(1) And when you run it, you get the same thing twice (whatever file was specified as the first argument to test-server.py on the commandline). On the other hand, change it to end print client.fetch(1) print client.fetch(1000) and you get "Hello World!" twice. At one point the server process also gave me this: Server(s) started. Press enter to s…

Thanks, just noticed a small initialization issue there, fixibg that soon :)

Re: Show HN: Framework for faster distributed data transfer

#6

Post by [deleted] »

kenko wrote: post #4

Change test-client.py to end: print client.fetch(1000) print client.fetch(1) And when you run it, you get the same thing twice (whatever file was specified as the first argument to test-server.py on the commandline). On the other hand, change it to end print client.fetch(1) print client.fetch(1000) and you get "Hello World!" twice. At one point the server process also gave me this: Server(s) started. Press enter to s…

[deleted]

Re: Show HN: Framework for faster distributed data transfer

#7

Post by andrewcooke »

wmf wrote: post #2

You said Luby, so prepare to be vaporized. On the technical side, theoretically this should only be a few percent faster than BitTorrent but requires significant complexity.

why vaporized?

also, why faster? does bittorrent duplicate some traffic?

Re: Show HN: Framework for faster distributed data transfer

#8

Post by wmf »

andrewcooke wrote: post #7
wmf wrote: post #2

You said Luby, so prepare to be vaporized. On the technical side, theoretically this should only be a few percent faster than BitTorrent but requires significant complexity.

why vaporized? also, why faster? does bittorrent duplicate some traffic?

BitTorrent has overhead from control traffic (have messages etc.) that can theoretically be eliminated using rateless coding. The extra decoding makes it a wash IMO.

Re: Show HN: Framework for faster distributed data transfer

#9

Post by evgen »

wmf wrote: post #8
andrewcooke wrote: post #7

Earlier quoted context omitted.

why vaporized? also, why faster? does bittorrent duplicate some traffic?

BitTorrent has overhead from control traffic (have messages etc.) that can theoretically be eliminated using rateless coding. The extra decoding makes it a wash IMO.

FWIW, when building the second version of the core tech from which BT was of forked we did license the tornado codecs from digital fountain to try to get a better reliability rate and it ended up being worse than a wash; setting aside the additional complexity it adds to the system it completely screws with internal block verification based on hashes. The blocks are also not structured compared to other FEC codecs (e.g. Rizzo) so you lose any hope of streaming. Long sorry short, unless you are trying to get around nasty packet loss there are better encoding schemes for this sort of data distribution task.
Post reply on HN