Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

11–20 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#11

Post by ratorx »

lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global.

In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except really popular downloads, Seeders don’t really check for fairness and will typically upload to whoever they can the fastest (with a limit on number of peers).

There’s an even more adversarial case because the unit of validation (a piece with a fixed hash from the spec) might be bigger than the chunks that are being shared individually. So it’s possible for a peer to fake having pieces and upload garbage data instead, and they wouldn’t be caught since different chunks came from different peers.

Re: Building a BitTorrent client from the ground up in Go (2020)

#12

Post by shellac »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

Most torrent clients I know support UPnP and/or NAT-PMP to request port forwarding from routers. As you say, it just isn't going to work for many people without this.

I run a BitTorrent service for an academic institution, to disseminate research data. We have a regular routable IP address, but still need to navigate the institutional firewall.

Re: Building a BitTorrent client from the ground up in Go (2020)

#13

Post by untitaker_ »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

the answer to this problem in general is NAT hole punching, but BitTorrent doesn't actually have an answer to your problem. if you are behind a NAT, you can only connect to peers that are not behind a NAT or have port forwarding set up. for popular torrents this is good enough because you don't have to connect to all peers.

> This is possible with port forwarding. But that's a niche set of peers, who have the power to configure port forwarding on a NAT proxy.

yes it's niche but I guess this means BitTorrent isn't as P2P in practice as one wants it to be, but held up by seedboxes.

Re: Building a BitTorrent client from the ground up in Go (2020)

#14

Post by pzmarzly »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

Most clients try to set up port forwarding using UPnP IGD, NAT PMP, or PCP protocol, a lot of residential routers support one of them.

If you enable WebTorrent as a transport protocol (enabled in gotorrent, disabled by default in libtorrent), it should be possible to use existing public STUN/TURN infrastructure, but I don't know if any client does it yet.

In practice, you just have to accept that many connections will simply fail, and make your client move on to try a different peer.

Re: Building a BitTorrent client from the ground up in Go (2020)

#15

Post by cybrox »

johnisgood wrote: post #7
ktpsns wrote: post #6

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

> Go is the go-to-language for complex networking situations And Elixir / Erlang for serious ones.

Rarely on the client side.

Re: Building a BitTorrent client from the ground up in Go (2020)

#16

Post by pkolaczk »

ktpsns wrote: post #6

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

> even async rust/python world would not look so clean as in Go.

That’s quite debatable and my experience is different. There is a whole lot of high level stuff that can be expressed with eg async streams and functional transformation chains in Rust, that Go has no counter offer for. Same for being able to use any future in select/join not just channels. Also I find cleanup / error handling in Rust much cleaner.

Re: Building a BitTorrent client from the ground up in Go (2020)

#17

Post by jedimastert »

lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download).

It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced

https://en.wikipedia.org/wiki/Leecher_(computing)

Re: Building a BitTorrent client from the ground up in Go (2020)

#18

Post by z0ltan »

pkolaczk wrote: post #16
ktpsns wrote: post #6

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

> even async rust/python world would not look so clean as in Go. That’s quite debatable and my experience is different. There is a whole lot of high level stuff that can be expressed with eg async streams and functional transformation chains in Rust, that Go has no counter offer for. Same for being able to use any future in select/join not just channels. Also I find cleanup / error handling in Rust much cleaner.

[dead]

Re: Building a BitTorrent client from the ground up in Go (2020)

#19

Post by neonsunset »

ktpsns wrote: post #6

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

And Go is not even the best or fastest language at this either - Goroutines are quite caveman of API, with footguns solved by better languages.

Re: Building a BitTorrent client from the ground up in Go (2020)

#20

Post by pzmarzly »

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - IIRC some clients measure average packet latency and if it starts going up, put some backpressure), resolve magnet URLs, set up port forwarding, reconfigure firewalls, offer API for *arr stack, be a good netizen (report stats correctly, send packets within the specs, do not spam - otherwise other clients will blocklist your in their code or config), implement many BEPs (mutable torrents are cool), be able to recover from interrupted state based on only the data that's on disk, have configurable downloading order (people want to start playing videos before they finish downloading, so you may want to e.g. download header and footer of each file first), and ideally detect duplicates between torrents (cross-seeding). And then there will be people throwing 2TB+ torrents at it (e.g. TLMC) to benchmark it, and saying your client is "literally unusable" if it doesn't handle it.

So, building a "perfect" torrent client from the ground up is a daunting task. But the "good" news is that nobody built such a "perfect" client just yet, so if you have some spare months of your time, you can take a shot at it. Or even better yet, open the issue tracker for one of popular clients or libraries, and add one of the missing features from the list above.

Post reply on HN