Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Station307 – Stream files between cURL, Wget and/or browser

station307.com

Post reply on HN
Print view View original

11–17 of 17 posts

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#11

Post by vortico »

I've recently wanted to write a streaming download service like this recently. The great thing about your service is that although it's a proof of concept of a handful of neat tricks, the core functionality works perfectly and could be extremely useful. This makes it faster than ever to drop upload links in a chat room and effectively solves the problem in https://xkcd.com/949/.

Is the backend licensed as free software? It would be fantastic for admins to host their own version of this since it allows the concept to propagate without worry of shutting down.

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#12

Post by hakvroot »

pkd wrote: post #10

Looks great! What is the technology stack behind this, if I may ask?

Thank you, and of course you may.

The backend is written in Java(8) and it is built with the following (noteworthy) components :

  Netty[1] - Extremely fast and stable Java networking framework,
             serves static files and relays traffic 
             (both tasks are custom implementations)

  Guice[2] - IoC container by Google, nice for keeping the code clean

  jOOQ[3] - I log some statistics in PostgreSQL 
            (file size, content type, transfer time - but no filenames). 
            If you're comfortable with SQL jOOQ is a real breeze to use.
The backend is hosted on the cheapest VM option at Digital Ocean (Amsterdam region), (which seems to be holding up pretty well so far :). I do run PostgreSQL on another $5 VM, so the Station307 server only has to worry about serving static content and moving large chunks of bytes around. SSL certificates are kindly provided by Let's Encrypt[4].

For the frontend, I know for a fact it uses left-pad (babel-core)... Written in JavaScript (ES2015 + JSX) and built with :

  Mithril[5] - Fast, small and simple React-like framework, quite a joy to use.

  Bootstrap[6] - Because not having to worry about L&F is quite nice. 
                 Trimmed it down a bit by only importing the parts I needed (SCSS).
And I think that's about it. If there's anything else you'd like to know, let me know!

  [1] http://netty.io
  [2] https://github.com/google/guice
  [3] http://www.jooq.org
  [4] https://letsencrypt.org
  [5] http://mithril.js.org
  [6] https://getbootstrap.com
edit: formatting D:

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#13

Post by hakvroot »

vortico wrote: post #11

I've recently wanted to write a streaming download service like this recently. The great thing about your service is that although it's a proof of concept of a handful of neat tricks, the core functionality works perfectly and could be extremely useful. This makes it faster than ever to drop upload links in a chat room and effectively solves the problem in https://xkcd.com/949/ . Is the backend licensed as free softw…

Currently I haven't licensed the backend as free software; the primary reason for this is that I reuse quite a lot of components from a different project of mine (https://gemin.ee) and I haven't decided yet on whether I want to release those components or remove them as dependencies. Since I too am familiar with useful stuff suddenly disappearing I'll give you my word that I will release the source if I have to shut down. If I decide to share the source I'll post a link to it on the site itself, so you won't have to look for it ;).

On another note, I'd encourage you to write a streaming download service. Compared to file uploading - storing - downloading I found it fun to do but also quite non-trivial to accomplish. I am planning to have another go at it myself, but with a different language (Go, Pony or Elixir). When run alongside of each other with the same UI/frontend/api it would make a nice benchmarking mechanism :).

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#14

Post by hakvroot »

maxpert wrote: post #9
hakvroot wrote: post #8

Author here! This is a little side project I created because I found myself using scp + email to send a file from a server to someone sitting right next to me more than once. I have a different project which does relaying over HTTP so I figured it might be a good idea to reuse a part of the codebase for a simple file streaming service. I finished it yesterday and it already proved useful for myself, and I hope it als…

Hey really nice to hear from you. Would be really nice if you share source code with the community. I really like the HTTP status code 100 trick :D

Thanks! I made some comments regarding to open sourcing it in reply to vortico (this thread) - current status is contemplating.

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#15

Post by darkf »

hakvroot wrote: post #7
exhilaration wrote: post #6

How does this work? I see the pending POST in the Chrome network tab, which completes when someone downloads it, and then become pending again for the next download. So are you proxying the uploaded data to the downloading client?

That's exactly what's going on. In a bit more detail: when a POST arrives and no receiver is available yet the server lets the sender know that it won't handle any more data (TCP Zero Window). When the receiver connects to the server the server asks the sender to continue writing and the data is relayed (proxied) from sender to receiver. If multiple receivers connect they're put into a queue and the server waits for…

Hi! Thanks a lot for sharing details about it, it's an ingenious app. :)

Could you please elaborate a bit on how you work with the TCP window size -- how might I set it to 0 and back? Do you simply use SO_RCVBUF or is there more to it?

If you happen to go through with making more variants of this, it would be amazing if you could make them open source as well (even just the backend). It's a clever idea that solves a very real problem!

Re: Show HN: Station307 – Stream files between cURL, Wget and/or browser

#17

Post by hakvroot »

darkf wrote: post #15
hakvroot wrote: post #7

Earlier quoted context omitted.

That's exactly what's going on. In a bit more detail: when a POST arrives and no receiver is available yet the server lets the sender know that it won't handle any more data (TCP Zero Window). When the receiver connects to the server the server asks the sender to continue writing and the data is relayed (proxied) from sender to receiver. If multiple receivers connect they're put into a queue and the server waits for…

Hi! Thanks a lot for sharing details about it, it's an ingenious app. :) Could you please elaborate a bit on how you work with the TCP window size -- how might I set it to 0 and back? Do you simply use SO_RCVBUF or is there more to it? If you happen to go through with making more variants of this, it would be amazing if you could make them open source as well (even just the backend). It's a clever idea that solves a…

Thank you! And I'd be happy to elaborate a bit more, although you already explained most of it yourself ;). Currently the server just stops reading from the socket and the rest is being handled by the networking stack (i.e. buffer fills up, window size is to 0). At this moment the buffer size is also fixed (it sits somewhere between having good throughput and not consuming too much memory for parked connections), but an improvement would be to change it dynamically to improve behavior on both fronts.

For any other versions or source releases I'll post an update!

Post reply on HN