Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a BitTorrent client from the ground up in Go (2020)

blog.jse.li

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Building a BitTorrent client from the ground up in Go (2020)

#21

Post by thrdbndndn »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

For me, the hardest part of building a good BT client (or any software, really) is getting the UX right.

These days, I use QB after uTorrent's downfall, but even after all these years, its UX still isn’t quite there.

Re: Building a BitTorrent client from the ground up in Go (2020)

#22

Post by jedimastert »

SoKamil wrote: post #5

This looks surprisingly simple. A perfect project to learn programming language by creating thing you can interact with and not get overwhelmed by domain complexity. Do you know more such problems?

There's an ad at the end of the post for something called CodeCrafters at the end of the post that appears to be a repository of such interesting problems

Re: Building a BitTorrent client from the ground up in Go (2020)

#23

Post by hoseja »

jedimastert wrote: post #17
lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

> Now I wonder how clients protect themselves against abusers (i.e. people who never upload a single bit but only download). It's called leeching, and it depends. It's typically considered good etiquette to upload as much as you download and that's usually enough, but it can be enforced https://en.wikipedia.org/wiki/Leecher_(computing)

I have the opposite problem, many things I download seem so well-seeded (seedboxes?) that I rarely achieve any sort of respectable ratio even if my uplink idles most of the time. It might be client problem but I haven't found much anyone discussing this. TAKE MY BITS!

Re: Building a BitTorrent client from the ground up in Go (2020)

#24

Post by TheDong »

SoKamil wrote: post #5

This looks surprisingly simple. A perfect project to learn programming language by creating thing you can interact with and not get overwhelmed by domain complexity. Do you know more such problems?

Creating a simple ray tracer for learning a language’s vector math libraries and parallelization tooling.

An FTP client and server.

An IRC client for the older protocol version.

A gemtext parser and gemini client.

A redis clone supporting only the simplest operations.

A brainfuck interpreter.

Twitter.

A todo list program (maybe a little too hard for this list)

Re: Building a BitTorrent client from the ground up in Go (2020)

#25

Post by Eikon »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> but very hard to make a good torrent client

> So, building a "perfect" torrent client from the ground up is a daunting task.

Don’t you think you’re exaggerating a bit? It’s not daunting by any stretch. The feature set you described is fairly straightforward and something even a beginner developer could tackle without too much hassle.

I’m honestly kind of tired of seeing people act like doing anything these days is some impossible feat.

Re: Building a BitTorrent client from the ground up in Go (2020)

#26

Post by rakoo »

lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

Sufficiently evolved torrent clients follow a tit-for-tat protocol:

- If you're nice with me I'll be nice with you

- If you're mean with me I'll be mean with you

This is the best answer for the Prisoner's dilemma (https://en.wikipedia.org/wiki/Prisoner's_dilemma) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate.

In bittorrent it's typically implemented as follows: peers start by sending a very minimal content, and see what the other replies. If they reply with low enough latency then slowly increase the amount of content that is sent and see if they reply with the same increased amount of content; if they do, continue up to the max of what the network link allows (in combination with other peers of course). If at some point the other peer doesn't send something equivalent (even though we know they have it and we asked for it) then that peer can be cut off for not being cooperative.

Situation is different for seeds of course, because they have everything and want nothing, but they can have a similar behaviour -- start sending a little, increase slowly over time

Re: Building a BitTorrent client from the ground up in Go (2020)

#27

Post by bool3max »

pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> But the "good" news is that nobody built such a "perfect" client just yet,

Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

Re: Building a BitTorrent client from the ground up in Go (2020)

#28

Post by andai »

bool3max wrote: post #27
pzmarzly wrote: post #20

It is easy to make a torrent client, but very hard to make a good torrent client. A very good, or let's say "perfect", one, needs to support multiple transport protocols (TCP, "uTP" aka UDP, "WebTorrent" aka WebRTC), multiple discovery mechanisms (DHT, PEX, HTTP trackers, WebSocket trackers), multiple torrent formats (v1, v2, hybrid), should use the network optimally (max the speed without overloading the network - I…

> But the "good" news is that nobody built such a "perfect" client just yet, Which of the qualities you listed are lacking from the currently most popular torrent clients (qBitTorrent, etc..)?

Personally, "being as small as uTorrent used to be", but clearly that's not a deal-breaker. (Then again, neither is like two thirds of the stuff GP mentioned!)

Re: Building a BitTorrent client from the ground up in Go (2020)

#29

Post by BonoboIO »

rakoo wrote: post #26
lqet wrote: post #2

Very cool, and surprisingly simple. I was always mystified as a kid how these file-sharing protocols actually work, but if you just look at a torrent as a collection of fixed-size pieces you are asking peers managed by a tracker for, it becomes really straightforward to implement. Now I wonder how clients protect themselves against abusers (i.e.. people who never upload a single bit but only download). I often notice…

Sufficiently evolved torrent clients follow a tit-for-tat protocol: - If you're nice with me I'll be nice with you - If you're mean with me I'll be mean with you This is the best answer for the Prisoner's dilemma ( https://en.wikipedia.org/wiki/Prisoner's_dilemma ) on the long run, ie a situation where peers don't trust each other but will both gain if both cooperate. In bittorrent it's typically implemented as follo…

Did knew, that the client does that with the tit-for-that.

In an ideal world the network would prioritize to upload to high speed like webseeds, so the webseeds can distribute even faster for everyone else.

Or do I make a mistake in my mind and the network already distributes efficiently and it is not needed.

Re: Building a BitTorrent client from the ground up in Go (2020)

#30

Post by miki123211 »

imbusy111 wrote: post #10

One thing that I find hard to understand is how you're able to connect to a random peer online with just an IP and port combination. The peer is likely behind a NAT proxy, and never communicated with you to begin with. In my head, NAT shouldn't allow this connection through, or be able to associate the port with the individual peer machine since the connection source (you) is arbitrary. This is possible with port for…

There are a couple ways to do this:

1. Manual port forwarding, it's likely that there are at least a few power users who already have the torrent and are seeding who have this set up. Seedboxes are a notable example, they're often simple servers that actually have a public IP.

2. UPNP, a protocol that lets you ask your router to set up a temporary port forward for you. Again, not all peers support this, but some do, and you can just connect to those.

3. Hole Punching. Imagine Alice is sending data to Bob, and her router ends up sending it from port 1234. Her router needs to send the packets it receives on port 1234 back to Alice's computer, to allow her to receive Bob's responses. Some routers will do this no matter which IP the packets are coming from. If Bob tells alice her router is sending from port 1234, she can spread that fact to others and let them contact her that way.

I don't know if BitTorrent clients take advantage of this specifically, but it's a very common way of doing NAT traversal in general.

NATs is why private trackers have the concept of "connectability", if you're "connectable", it means you can accept connections from other clients. Crucially, if just one of the peers is connectable, they can both communicate, so connectability is heavily encouraged but not required.

Post reply on HN