Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

31–40 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#31

Post by pzmarzly »

bool3max wrote: post #27
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

I am using qBittorrent, and from the top of my head:

- WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision),

- cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own)

- when it comes to torrent management, there is no way to group torrents into groups with common settings (important if you use multiple private trackers) - people recommend having multiple installations of qBittorrent side by side

- when it comes to reconfiguring NAT and firewall, qBittorrent supports UPNP IGD protoocl, but I am not sure about NAT-PMP and PCP

- I have never seen qBittorrent connect to a single IPv6 peer - so I don't know if the support is there

- download order - you can choose "download in order" or "download rarest first". I dont think "download in order" downloads footers, so mp4 files won't work (IIRC mp4 store metadata in footer, mkv in header)

Re: Building a BitTorrent client from the ground up in Go (2020)

#32

Post by troupo »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

You may also want both a CLI and a web access/management interface

Re: Building a BitTorrent client from the ground up in Go (2020)

#33

Post by andai »

rakoo wrote: post #26
lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

Sufficiently evolved torrent clients follow a tit-for-tat protocol: - If you're nice with me I'll be nice with you - If you're mean with me I'll be mean with you This is the best answer for the Prisoner's dilemma ( https://en.wikipedia.org/wiki/Prisoner's_dilemma ) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate. In bittorrent it's typically implemented as follo…

> [tit for tat] is the best answer for the Prisoner's dilemma on the long run

This is fascinating. Have there been any simulations about this? I'm sure they've looked into this in game theory, but I'm wondering if you have a big swarm of torrent clients, does the scale of the population change the outcome, or is it the same?

(It seems like it would be wise when designing the torrent clients to run such simulations, so I'm thinking this probably has been done at some point.)

Re: Building a BitTorrent client from the ground up in Go (2020)

#34

Post by pzmarzly »

andai wrote: post #28
bool3max wrote: post #27

Earlier quoted context omitted.

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

Personally, "being as small as uTorrent used to be", but clearly that's not a deal-breaker. (Then again, neither is like two thirds of the stuff GP mentioned!)

To be fair, circa 2010 all torrent clients got "good enough", so my wishlist above is just a wishlist (because torrents are cool and I like to see progress). I use (and sponsor) qBittorrent myself and have no plans to change.

Re: Building a BitTorrent client from the ground up in Go (2020)

#35

Post by pzmarzly »

thrdbndndn wrote: post #21
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

For me, the hardest part of building a good BT client (or any software, really) is getting the UX right. These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.

qBittorrent + VueTorrent is not perfect, but it's hard to find anything better. If you haven't tried it, take a look.

https://github.com/VueTorrent/VueTorrent

Re: Building a BitTorrent client from the ground up in Go (2020)

#36

Post by Akronymus »

hoseja wrote: post #23
jedimastert wrote: post #17

Earlier quoted context omitted.

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download). It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced https://en.wikipedia.org/wiki/Leecher_(computing)

I have the opposite problem, many things I download seem so well-seeded (seedboxes?) that I rarely achieve any sort of respectable ratio even if my uplink idles most of the time. It might be client problem but I haven't found much anyone discussing this. TAKE MY BITS!

Over here (austria), you are pretty free to leech as much as you want, but as soon as you actually start seeding at any capacity, people start paying very close attention.

Re: Building a BitTorrent client from the ground up in Go (2020)

#37

Post by wslh »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Yes, even qBitorrent "destroys" your other connection at maximum speeds.

Re: Building a BitTorrent client from the ground up in Go (2020)

#38

Post by dataviz1000 »

johnisgood wrote: post #7
ktpsns wrote: post #6

This looks simple not only because the article is written well but also because Go is the go-to-language for complex networking situations. Doing things in parallel, even pipelining? This would make quite some spaghetti algorithm in C/C++, even async rust/python world would not look so clean as in Go. This is clearly a big strength of the language.

> Go is the go-to-language for complex networking situations And Elixir / Erlang for serious ones.

Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. It's great when starting out with a few developers, however, since Go and Rust are much more performant, it is possible to hire several more Go or Rust developers with the server cost savings instead of being tied to Elixir. There is always a trade off between easy to use and high performance.

But, yeah, if I was going to bootstrap a startup at the seed level, Elixir is the best choice for backend. If I'm spending $500K+ a year on infrastructure, I'll be looking at Go and Rust.

Re: Building a BitTorrent client from the ground up in Go (2020)

#39

Post by szvsw »

thrdbndndn wrote: post #21
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

For me, the hardest part of building a good BT client (or any software, really) is getting the UX right. These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.

As someone who stopped torrenting around 2015 or maybe 2016, what happened to uTorrent? Also RIP what.cd, I still am sad that it is gone.

Re: Building a BitTorrent client from the ground up in Go (2020)

#40

Post by 6510 »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.
Post reply on HN