Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

51–60 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#51

Post by anacrolix »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

The other answers are correct about only needing a single peer among any pair of peers to be addressable.

However there is a built-in hole punching mechanism in BitTorrent where peers ask for a third peer to assist in hole punching.

It's implemented in my client. It was very painful to implement. I think someone privately funded the feature which was very nice.

Re: Building a BitTorrent client from the ground up in Go (2020)

#52

Post by anacrolix »

newsclues wrote: post #44
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Given the state of software quality, it is remarkable that torrent clients are SO GOOD, given they are free!

There's only a handful of actually complete ones. A few it's surprising they even work, they're crazy complicated, and weird, but once they're stable they need very little maintenance. They're all also written in very different styles due to a mix of C/C++ and very high concurrency.

Re: Building a BitTorrent client from the ground up in Go (2020)

#53

Post by anacrolix »

6510 wrote: post #40
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

Alternatively, a minimal client with few features that can seed large numbers of torrents without crashing or freezing. Settings for how many it may seed at a time, prioritize that what needs seeding, ignore overly seeded things. Options to seed only blocks no one has. Remotely controlled.

I had a super seeder implementation that seeded over 400k torrents simultaneously using some fancy callbacks at certain points in the protocol.

If this is actually a common use case, I can resurrect it into a usable form for the public if there's interest or funding. https://github.com/anacrolix/torrent

Re: Building a BitTorrent client from the ground up in Go (2020)

#54

Post by wincy »

bool3max wrote: post #27
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

Last I tried it their hybrid torrent implementation was very broken and would fail half the time.

Re: Building a BitTorrent client from the ground up in Go (2020)

#55

Post by __MatrixMan__ »

hoseja wrote: post #23
jedimastert wrote: post #17

Earlier quoted context omitted.

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download). It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced https://en.wikipedia.org/wiki/Leecher_(computing)

I have the opposite problem, many things I download seem so well-seeded (seedboxes?) that I rarely achieve any sort of respectable ratio even if my uplink idles most of the time. It might be client problem but I haven't found much anyone discussing this. TAKE MY BITS!

I've often considered writing the equivalent of a robotrader for this. If you're paying for a sedbox and want to use it to the fullest re: maximizing your ratio then some automated attempt to predict demand and start seeding the right thing early would likely go a long way.

Unlike predicting the stock market, competition wouldn't be very fierce, and I think the tracker API would give you most of what you need.

It's in the long list of things I'll do if I ever find a big pile of time.

Re: Building a BitTorrent client from the ground up in Go (2020)

#56

Post by anacrolix »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

There's a less commonly known feature of NAT (maybe called cone NAT or something), where the remote endpoints are not checked for incoming packets and connections. You communicate with trackers and DHT and as usual your outbound packets are mapped to some port. But everyone use sees your public IP and port and talk about your client using that pair. They then also communicate with you over UDP to your public port. As long as you have regular outbound traffic running through that mapping, NAT will keep the hole alive. I think with testing I found at least half of people on NAT had this. It's less common on mobile and fibre.

Re: Building a BitTorrent client from the ground up in Go (2020)

#57

Post by pzmarzly »

anacrolix wrote: post #45
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols. I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed. I also have all the DHT extensions, like mutable torrents and get/put etc. Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue…

I know your library does a lot, I have been your GH Sponsor for months ;) Thanks for reminding me about it, I wanted to up the amount. And to thank you for working on such cool stuff and making the ecosystem better.

I missed the upnp support, thanks for correcting my beliefs.

Which client with WebUI would you recommend the most? I tried using exatorrent and distribyted, but was running into bugs.

Re: Building a BitTorrent client from the ground up in Go (2020)

#58

Post by silotis »

pzmarzly wrote: post #31
bool3max wrote: post #27

Earlier quoted context omitted.

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into…

> NAT-PMP and PCP

qBittorrent (via libtorrent) supports NAT-PMP and PCP

> IPv6

qBittorrent supports IPv6

Re: Building a BitTorrent client from the ground up in Go (2020)

#59

Post by WithinReason »

pzmarzly wrote: post #31
bool3max wrote: post #27

Earlier quoted context omitted.

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into…

qbittorrent has "download first and last pieces first" (in addition to "download in order") so the footer is retrieved

Re: Building a BitTorrent client from the ground up in Go (2020)

#60

Post by kamranjon »

I was just thinking about how incredible BitTorrent was the other day, how it really does outperform non-distributed solutions in many cases and how it is one of the rare examples of that.

One thing I was thinking though was that the finding of Peers seems to be the bottleneck - that if peer resolution could be almost instantaneous, the BT protocol could be used for so many more use cases.

Does anyone know if this part of the process could ever be improved or does that just come with the territory?

Post reply on HN