Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

81–90 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#81

Post by a-french-anon »

anacrolix wrote: post #45
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols. I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed. I also have all the DHT extensions, like mutable torrents and get/put etc. Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue…

Any clean room impl. is good news to me, but is the support for private torrent/tracker complete (https://github.com/anacrolix/torrent/issues/531)?

NB: especially after https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).

Re: Building a BitTorrent client from the ground up in Go (2020)

#82

Post by johnisgood »

cultofmetatron wrote: post #78
dataviz1000 wrote: post #38

Earlier quoted context omitted.

Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. It's great when starting out with a few developers, however, since Go and Rust are much more performant, it is possible to hire several more Go or Rust developers with the server cost savings instead of being tied to Elixir. There is always a trade off between easy to use and high performance. But, yeah, if I was going to bootstrap…

> Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. for cpu bound tasks? sure. but we are talking in the context of networking. elixir is going to absolutely smoke go for applications requiring a lot of simultaneous connections. We can already see it in actionable vs phoenix channels. channels supports a magnitude order more simultaneous websocket connections per machine. ps: libr…

And for CPU-bound tasks one could use NIFs (usually better) or Ports if absolutely necessary.

https://stackoverflow.com/questions/42035912/running-c-code-...

Re: Building a BitTorrent client from the ground up in Go (2020)

#83

Post by pdimitar »

Eikon wrote: post #25
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> but very hard to make a good torrent client > So, building a "perfect" torrent client from the ground up is a daunting task. Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle. I’m honestly kind of tired of seeing people act like doing anything these days is…

Why would it be an exaggerated claim?

Try it yourself, from scratch, and see how quickly you will finish something that will rival and surpass, say, qBittorrent.

None of the tasks is breaking new ground or is creatively difficult. But there's a metric ton of those tasks in order to make a truly good torrent client.

Re: Building a BitTorrent client from the ground up in Go (2020)

#84

Post by pzmarzly »

SSLy wrote: post #48
pzmarzly wrote: post #31

Earlier quoted context omitted.

I am using qBittorrent, and from the top of my head: - WebTorrent and WebSocket patch for qBittorrent is ready but not merged (waiting on libtorrent's decision), - cross-seeding support is poor (a separate "cross-seed" binary can be used to set up hardlinks to fool qBittorrent into cross-seeding, but it cannot detect duplicates on its own) - when it comes to torrent management, there is no way to group torrents into…

> NAT-PMP https://github.com/arvidn/libtorrent/blob/c31d90b59ffb3910b6... > PCP seems like vaporware

Isn't PCP the only protocol of these 3 that can reconfigure IPv6 firewall?

Re: Building a BitTorrent client from the ground up in Go (2020)

#85

Post by transmission3 »

a-french-anon wrote: post #81
anacrolix wrote: post #45

Earlier quoted context omitted.

My client anacrolix/torrent has most of that, including streaming, hybrid, v2, all trackers and protocols. I don't do much port forwarding, it's too fiddly. I think I provide upnp out of the box and few other variants that were contributed. I also have all the DHT extensions, like mutable torrents and get/put etc. Large torrents do pop up occasionally, but it's been a long time since someone found a performance issue…

Any clean room impl. is good news to me, but is the support for private torrent/tracker complete ( https://github.com/anacrolix/torrent/issues/531 )? NB: especially after https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).

another person still using transmission 3 here! no clue what i'm going to do if/when it stops compiling.

Re: Building a BitTorrent client from the ground up in Go (2020)

#86

Post by dataviz1000 »

cultofmetatron wrote: post #78
dataviz1000 wrote: post #38

Earlier quoted context omitted.

Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. It's great when starting out with a few developers, however, since Go and Rust are much more performant, it is possible to hire several more Go or Rust developers with the server cost savings instead of being tied to Elixir. There is always a trade off between easy to use and high performance. But, yeah, if I was going to bootstrap…

> Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. for cpu bound tasks? sure. but we are talking in the context of networking. elixir is going to absolutely smoke go for applications requiring a lot of simultaneous connections. We can already see it in actionable vs phoenix channels. channels supports a magnitude order more simultaneous websocket connections per machine. ps: libr…

Elixir lacks the fine grain control for optimizing I/O which Go provides. Moreover, a bit torrent client requires CPU intensive hash verification, encryption/decryption, and data compression/decompression which will greatly slow down Elixir. You can build a connector to Rust or instead of using slow abstraction layers you can build the thing in Rust if you have the skills.

Rather then tell me, you are going to have to build a bit torrent client in Elixir and show me. Otherwise, after my years experience working with an Elixir team, I don't believe you.

Re: Building a BitTorrent client from the ground up in Go (2020)

#87

Post by vjerancrnjak »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

All of that without encryption. What a meh

But shouldn’t be hard to do with modern facilities (async runtimes, streaming libraries etc.) I don’t enjoy many clients yet I use them.

It’s definitely a good problem with which to test a programming language. Many fail on parsing, or inefficient network interfaces, inefficient file interfaces, sha sum unusable without intrinsics, etc.

Re: Building a BitTorrent client from the ground up in Go (2020)

#88

Post by [deleted] »

dataviz1000 wrote: post #86
cultofmetatron wrote: post #78

Earlier quoted context omitted.

> Unfortunately, BEAM VM is very slow compared to multiple other languages including Go. for cpu bound tasks? sure. but we are talking in the context of networking. elixir is going to absolutely smoke go for applications requiring a lot of simultaneous connections. We can already see it in actionable vs phoenix channels. channels supports a magnitude order more simultaneous websocket connections per machine. ps: libr…

Elixir lacks the fine grain control for optimizing I/O which Go provides. Moreover, a bit torrent client requires CPU intensive hash verification, encryption/decryption, and data compression/decompression which will greatly slow down Elixir. You can build a connector to Rust or instead of using slow abstraction layers you can build the thing in Rust if you have the skills. Rather then tell me, you are going to have t…

[deleted]

Re: Building a BitTorrent client from the ground up in Go (2020)

#89

Post by a-french-anon »

transmission3 wrote: post #85
a-french-anon wrote: post #81

Earlier quoted context omitted.

Any clean room impl. is good news to me, but is the support for private torrent/tracker complete ( https://github.com/anacrolix/torrent/issues/531 )? NB: especially after https://github.com/Luminarys/synapse died and Transmission "botched" its 4.x rewrite (currently on 3 with some backported patches).

another person still using transmission 3 here! no clue what i'm going to do if/when it stops compiling.

https://github.com/stefantalpalaru/transmission-og

Re: Building a BitTorrent client from the ground up in Go (2020)

#90

Post by rakoo »

BonoboIO wrote: post #29
rakoo wrote: post #26

Earlier quoted context omitted.

Sufficiently evolved torrent clients follow a tit-for-tat protocol: - If you're nice with me I'll be nice with you - If you're mean with me I'll be mean with you This is the best answer for the Prisoner's dilemma ( https://en.wikipedia.org/wiki/Prisoner's_dilemma ) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate. In bittorrent it's typically implemented as follo…

Did knew, that the client does that with the tit-for-that. In an ideal world the network would prioritize to upload to high speed like webseeds, so the webseeds can distribute even faster for everyone else. Or do I make a mistake in my mind and the network already distributes efficiently and it is not needed.

A single node distributing everything "because it's fast" isn't really the architecture that bittorrent is aiming for, and I don't think it's one we should be aiming for.

Try downloading any well distributed torrent and you'll see your bandwidth capacity automatically maxxed out. The gradual decision with an emerging selection of better sources always leads to that. Bittorrent doesn't really have an efficiency problem

Post reply on HN