Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Rqbit – bittorent CLI and desktop app written in Rust

github.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Show HN: Rqbit – bittorent CLI and desktop app written in Rust

#11

Post by ikatson »

mixmastamyk wrote: post #10

How does one learn how to develop BT clients and servers? Or use libBT at a competent level? Every time I’ve looked for a book or in-depth tutorials I get swamped with user articles.

For rqbit at least, there were only a few things necessary to start off with:

1. The bittorrent protocol specification: https://www.bittorrent.org/beps/bep_0003.html

2. Wireshark dumps of some existing BitTorrent clients to write unit-tests for RPC serialization/deserialization. I used qBittorrent, but you can use any other existing client.

3. (kind of optional) DHT protocol: https://www.bittorrent.org/beps/bep_0005.html. This actually came later, you can download torrents using just #1. But if you try to do so, you'll discover that most peer information is stored in DHT, and not in trackers.

Everything else was heuristics, observing real network behaviour, and tweaking the code accordingly.

That said, I'm not exactly the go-to expert on "how to develop BT clients and servers", as rqbit isn't as fully featured as the more mature clients. But given that the above links got me that far, I'm sure they can give a very decent start.

Re: Show HN: Rqbit – bittorent CLI and desktop app written in Rust

#12

Post by anacrolix »

ikatson wrote: post #6
capableweb wrote: post #2

> I was not satisfied with my regular bittorrent client, and was wondering how much work would it be to create a new one from scratch, and it got where it is, starting from bencode protocol implemenation, then peer protocol, etc, etc. That's pretty cool :) Usually people just pull in libraries for all those things, so pretty nice you didn't, makes for more interesting code when you're building stuff just for fun! Of…

When I built it at first a couple years ago, it all went pretty smooth until I hit concurrent communication with many peers, both for DHT and torrent downloading. E.g. parsing bencode, and other binary protocols involved in the network (bittorrent peer protocol, tracker requests, DHT protocol) was all easy comparing to managing state. Managing state (e.g. what parts we have downloaded so far, what are we downloading…

I found the trickiest parts to be DHT concurrency (finally solved this adequately after 5-6 years of experimenting), and efficient block requesting (I've rewritten this every few years, but my latest implementation seems to be solid since 2020 or so). The major thing I've not solved, but also been too lazy to tackle is a peer cache. I just reaannounce when all peers are exhausted and start over.

Re: Show HN: Rqbit – bittorent CLI and desktop app written in Rust

#13

Post by ikatson »

anacrolix wrote: post #12
ikatson wrote: post #6

Earlier quoted context omitted.

When I built it at first a couple years ago, it all went pretty smooth until I hit concurrent communication with many peers, both for DHT and torrent downloading. E.g. parsing bencode, and other binary protocols involved in the network (bittorrent peer protocol, tracker requests, DHT protocol) was all easy comparing to managing state. Managing state (e.g. what parts we have downloaded so far, what are we downloading…

I found the trickiest parts to be DHT concurrency (finally solved this adequately after 5-6 years of experimenting), and efficient block requesting (I've rewritten this every few years, but my latest implementation seems to be solid since 2020 or so). The major thing I've not solved, but also been too lazy to tackle is a peer cache. I just reaannounce when all peers are exhausted and start over.

Concurrency and the shared state nature of torrents are definitely what makes it all hard and tricky. I've rewritten this part of DHT completely recently, but according to your experience looks like this time won't be the last.

For block requesting, rqbit has a pretty simple algorithm https://github.com/ikatson/rqbit/blob/main/crates/librqbit/s..., and I didn't notice it in benchmarks, thanks to Rust being fast by default I guess. I admit though, never looked how other clients do it, maybe the rqbit algorithm is too naive.

Re: Show HN: Rqbit – bittorent CLI and desktop app written in Rust

#14

Post by mixmastamyk »

ikatson wrote: post #11
mixmastamyk wrote: post #10

How does one learn how to develop BT clients and servers? Or use libBT at a competent level? Every time I’ve looked for a book or in-depth tutorials I get swamped with user articles.

For rqbit at least, there were only a few things necessary to start off with: 1. The bittorrent protocol specification: https://www.bittorrent.org/beps/bep_0003.html 2. Wireshark dumps of some existing BitTorrent clients to write unit-tests for RPC serialization/deserialization. I used qBittorrent, but you can use any other existing client. 3. (kind of optional) DHT protocol: https://www.bittorrent.org/beps/bep_0005.…

Thank you.

Re: Show HN: Rqbit – bittorent CLI and desktop app written in Rust

#15

Post by joshka »

mixmastamyk wrote: post #10

How does one learn how to develop BT clients and servers? Or use libBT at a competent level? Every time I’ve looked for a book or in-depth tutorials I get swamped with user articles.

Take a watch of Jon Gjengset's vids on this (there's a codecrafters challenge for building out a bittorent protocol that he's going through).

- https://www.youtube.com/watch?v=jf_ddGnum_4

- https://www.youtube.com/watch?v=r0srf3kfZbs

Post reply on HN