Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database-less torrent website

boredcaveman.xyz

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Database-less torrent website

#31

Post by tecleandor »

etagate wrote: post #13
tecleandor wrote: post #9

I'm curious about why didn't they compress the SQLite file using gzip. Even in level 1, for superfast compression/decompression halved the file size.

I knew I could do it, I just wanted to make a demo of the whole thing to see if it would have worked in practice. But yes there's space for a lot of improvements like compression and lazy load with sqlite (cited in a comment)

Great! I was honestly wondering because I didn't know if there were any technical limitations/problems, but it's quite interesting nonetheless.

Re: Database-less torrent website

#32

Post by oscargrouch »

lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

My current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem.

The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end.

So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated and the torrent "turns into another", the solution is to use the RPC api (that my solution provides) over the initial swarm of peers and use some sort of distributed algorithm like raft or gossip to define how to deal with the subsequent changes over the peers.

The important part is that all other changed torrents from all the peers are available to all peers in the network, and with a RPC working as a abstract interface to organize them, it will work for a lot of different scenarios.

Re: Database-less torrent website

#33

Post by rakoo »

lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

Mutable torrents do exist (https://www.bittorrent.org/beps/bep_0046.html). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.

Re: Database-less torrent website

#34

Post by ineedasername »

The headline doesn't make much sense. There's still a database, what changes is simply where it's stored when in use. "Torrent Website w/o DB Server" might work better.

That aside, would this method scale? 135,000 torrents doesn't seem comprehensive, so I would expect real world use to have many more. Maybe a different SQLite db for different categories?

Re: Database-less torrent website

#35

Post by smallerfish »

oscargrouch wrote: post #32
lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

My current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem. The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end. So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated…

That's interesting. Are you aware of the absurd-sql project? It may have some useful stuff that you can leverage.

Also, how do peers find each other? And finally, do you intend to open source it?

Re: Database-less torrent website

#36

Post by ThalesX »

vindarel wrote: post #2

related: https://torrent-paradise.ml/ Also based on IPFS with this HTTP gateway. I find a lot of torrents in there.

If any dev is here, I'm on IPFS and can't seem to search, getting console error:

> Uncaught ReferenceError: passQueryToResultpage is not defined

Re: Database-less torrent website

#37

Post by rdubz »

vermilingua wrote: post #23
danka wrote: post #10

I wonder why the author ignored the option of compression in the post. Even with a simple gzip DEFLATE compression, those 10MB of plain text could get as small as a 1MB archive and possibly more, meaning that in a compressed 10MB payload you could fit much much more than 135K records.

It isn't 10MB of plain text though, it's 10MB of binary SQLite database. I agree that compression would be useful here, but I don't think a simple gzip DEFLATE would be. I was curious so I compressed that torrent db with a few different methods: 11.1MB 11116544B dump.sqlite 10.2MB 10155419B dump.csv 6.6MB 6573399B dump.sqlite.gz 6.6MB 6565771B dump.zip 5.6MB 5616842B dump.rar gzip is certainly suitable to be used in…

Back of envelope:

The 10MB estimated size came from [100 bytes per row] * [100k rows].

50 of the bytes per row were "description", which should compress well (2-3x, I'd guess).

40 bytes per row were the IPFS ID/hash, IIUC. I assumed this is like a Git hash, 40 hex chars, which is really just 20 bytes of entropy.

He also estimated 14 bytes for the size (stored as a string representation of a decimal integer, up to 1e15 - 1, or 1PB?). That's about 50 bits or 6-7 bytes, as a binary integer. Sizes wouldn't be uniformly distributed though so it would compress to even fewer bytes.

So if SQLite was smart (or one gzips the whole db file, like you did), it makes sense that a factor of 2 or so is reclaimable.

Re: Database-less torrent website

#38

Post by mavhc »

Gigachad wrote: post #17
etagate wrote: post #15

Earlier quoted context omitted.

This is true even for a torrent btw. And (un?)fortunately you can distribute anything on IPFS as long as it's encrypted.

No one is going to pin encrypted data they do not understand. And IPFS doesn’t get users to seed random files, you only seed stuff you downloaded. So really it’s not much different to hosting it on a personal http server except any downloaders can also host.

Sounds very similar to bittorrent itself

Re: Database-less torrent website

#39

Post by ddtaylor »

rakoo wrote: post #33
lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

Mutable torrents do exist ( https://www.bittorrent.org/beps/bep_0046.html ). Your "URL" can now be a public key, and clients will fetch the latest version of the torrent.

This is actually really neat I wasn't aware of these BEPs. Do you know of any place where they are being used?

Re: Database-less torrent website

#40

Post by jtbayly »

donkarma wrote: post #28

That's not going to stop takedowns, they don't care about if it's technically on your site or not.

Exactly. I forget where I just saw this. It seemed like a particularly absurd example, but the examples are numerous. Torrent sites don’t have the copyrighted content either, but they get shut down all the time.
Post reply on HN