Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#61

Post by __MatrixMan__ »

ratorx wrote: post #11
lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global. In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except…

Wouldn't the peer uploading garbage data be caught immediately because the garbage wouldn't hash to the correct value? I imagine most clients would stop talking to such a peer rather quickly.

Re: Building a BitTorrent client from the ground up in Go (2020)

#62

Post by HumblyTossed »

TheDong wrote: post #24
SoKamil wrote: post #5

This looks surprisingly simple. A perfect project to learn programming language by creating thing you can interact with and not get overwhelmed by domain complexity. Do you know more such problems?

Creating a simple ray tracer for learning a language’s vector math libraries and parallelization tooling. An FTP client and server. An IRC client for the older protocol version. A gemtext parser and gemini client. A redis clone supporting only the simplest operations. A brainfuck interpreter. Twitter. A todo list program (maybe a little too hard for this list)

Oh wow. I remember all the "I wrote a twitter in a weekend. Why does twitter need a thousand engineers?" posts everywhere.

Re: Building a BitTorrent client from the ground up in Go (2020)

#63

Post by blacklion »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Also, good disk I/O scheduler needed, or you can thrash even stripped NVMe disks, not to mention spinning rust.

Good multi threading without any global locks is a must or your UI/remote control become unresponsive on big number of active torrents.

Also, it needs daemon mode, which can be compiled & installed without any GUI libs (like QT, GTK, Waynald, libx11, anything like this) and fast responsive remote GUI for all platforms which allows to manage thousands of torrents simultaneously (I'm looking at you, ). It could be Web UI, but for now I didn't se usable one. Only thing that semi-works is Transmission + Transmission-Remote-GUI written in Pascal. But it is effectively abadonware and works worse and worse with each Transmission update.

And be scriptable to allow automate change of torrent on tracker, for example (you need to detect it by "Torrent not registered" tracker-specific error answer and re-download torrent building URL from, for example, "comment" field of torrent, again in tracker-specific way).

Re: Building a BitTorrent client from the ground up in Go (2020)

#64

Post by dgroshev »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Few more challenges:

- seeding a million active torrents (this is entirely possible and needed but drives a lot of architecture)

- a UI for a million active torrents

- be nice to trackers (keep-alive, batched reports; again, architecture gets tricky, iirc libtorrent can't do that still)

- correct pipelining

- congestion control on uTP

- I/O scheduling/batching

- testing all of the above

It is quite a lot.

Re: Building a BitTorrent client from the ground up in Go (2020)

#65

Post by blacklion »

dgroshev wrote: post #64
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Few more challenges: - seeding a million active torrents (this is entirely possible and needed but drives a lot of architecture) - a UI for a million active torrents - be nice to trackers (keep-alive, batched reports; again, architecture gets tricky, iirc libtorrent can't do that still) - correct pipelining - congestion control on uTP - I/O scheduling/batching - testing all of the above It is quite a lot.

Looks like we have same experience with torrents :-)

Re: Building a BitTorrent client from the ground up in Go (2020)

#66

Post by dgroshev »

Eikon wrote: post #25
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> but very hard to make a good torrent client > So, building a "perfect" torrent client from the ground up is a daunting task. Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle. I’m honestly kind of tired of seeing people act like doing anything these days is…

It's not impossible, it's just hard to be better than libtorrent, otherwise a new client only makes sense as an exercise.

Arvid Norberg is hardly a "beginner developer", yet look how hard it is to make a performant implementation of uTP: https://github.com/arvidn/libtorrent/issues/3542

Re: Building a BitTorrent client from the ground up in Go (2020)

#68

Post by ratorx »

__MatrixMan__ wrote: post #61
ratorx wrote: post #11

Earlier quoted context omitted.

IIRC, when both peers only have some subset of pieces, they are meant to “trade” with each other, and so will eventually drop low value peers for higher value ones. Aside from the adversarial case, this helps optimise for being nearer to the people you are sharing with, since torrents are global. In practice, BitTorrent really needs seeders who have downloaded the entirety of the file to be fast for everything except…

Wouldn't the peer uploading garbage data be caught immediately because the garbage wouldn't hash to the correct value? I imagine most clients would stop talking to such a peer rather quickly.

Torrents are split into pieces, each of which has a hash that allows it to be validated independently.

However, these pieces can themselves be large, so it is often recommended to split them into smaller chunks that are sent one at a time. You cannot validate a chunk independently, and if you assemble a piece from chunks from multiple origins, you can’t immediately tell which origin is bad.

You need to explicitly handle this case, like the sibling comment mentioned by isolating peers who may have sent bad data and forcing validation.

Re: Building a BitTorrent client from the ground up in Go (2020)

#69

Post by 6510 »

anacrolix wrote: post #53
6510 wrote: post #40

Earlier quoted context omitted.

Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.

I had a super seeder implementation that seeded over 400k torrents simultaneously using some fancy callbacks at certain points in the protocol. If this is actually a common use case, I can resurrect it into a usable form for the public if there's interest or funding. https://github.com/anacrolix/torrent

https://www.reddit.com/r/DataHoarder/comments/wcusy1/torrent...

The datahoarders run multiple clients. It's a workable solution but far from ideal.

Re: Building a BitTorrent client from the ground up in Go (2020)

#70

Post by Marsymars »

pzmarzly wrote: post #31
bool3max wrote: post #27

Earlier quoted context omitted.

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into…

> - when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side

Probably easiest to do this via docker-qbittorrent-nox.

> - I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there

The Linux ISOs (not a euphemism) I'm seeding probably get about a third of their peer connections via IPv6.

Post reply on HN