Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Database-less torrent website

boredcaveman.xyz

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Database-less torrent website

#21

Post by samwillis »

yonixw wrote: post #4

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630

As the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent?

(My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api)

EDIT:

After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not.

https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...

EDIT2:

In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too.

https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...

Re: Database-less torrent website

#22

Post by dfghdfhs »

Could someone more savvy clarify....

On that page I do inspect, got console and do:

loadDBAndExec('SELECT * FROM my_table LIMIT 10;')

I get

Uncaught ReferenceError: loadDBAndExec is not defined

Why doesn't this work?

Re: Database-less torrent website

#23

Post by vermilingua »

danka wrote: post #10

I wonder why the author ignored the option of compression in the post. Even with a simple gzip DEFLATE compression, those 10MB of plain text could get as small as a 1MB archive and possibly more, meaning that in a compressed 10MB payload you could fit much much more than 135K records.

It isn't 10MB of plain text though, it's 10MB of binary SQLite database. I agree that compression would be useful here, but I don't think a simple gzip DEFLATE would be.

I was curious so I compressed that torrent db with a few different methods:

  11.1MB 11116544B dump.sqlite
  10.2MB 10155419B dump.csv
   6.6MB  6573399B dump.sqlite.gz
   6.6MB  6565771B dump.zip
   5.6MB  5616842B dump.rar
gzip is certainly suitable to be used in this situation, I stand corrected.

Re: Database-less torrent website

#25

Post by manigandham »

dfghdfhs wrote: post #22

Could someone more savvy clarify.... On that page I do inspect, got console and do: loadDBAndExec('SELECT * FROM my_table LIMIT 10;') I get Uncaught ReferenceError: loadDBAndExec is not defined Why doesn't this work?

That's just example code in the blog post.

The live demo uses a webworker with different code: https://boredcaveman.xyz/demo/megacat/database-app.js

Re: Database-less torrent website

#26

Post by gerwim »

lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

Well there is Dolt: https://www.dolthub.com/

Re: Database-less torrent website

#27

Post by mosselman »

lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

> (or, updates require re-deployment of full new .sqlite db, together with a new website)

Seems like a small price to pay.

I can see new stuff living in torrents that then get committed to a sqlite database that is then deployed every now and then.

Re: Database-less torrent website

#29

Post by samwillis »

samwillis wrote: post #21
yonixw wrote: post #4

Enhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630

As the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent? (My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api) EDIT: After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on…

Continuing my looking into this, the SQLite range requests trick was implemented in this [0] project.

It turns out that project was inspired by a few others [1], two of which [2][3] implement a vfs for SQLite on top of bittorrent doing exactly this suggestion, but with a bittorrent hosted file rather than IPFS. They only download the parts of the SQLite files needed when querying.

0: https://github.com/phiresky/sql.js-httpvfs

1: https://github.com/phiresky/sql.js-httpvfs#inspiration

2: https://github.com/lmatteis/torrent-net

3: https://github.com/bittorrent/sqltorrent

Re: Database-less torrent website

#30

Post by chrisjc »

lekevicius wrote: post #20

One unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?

> Something that would have immutable rows

Isn't this just a log (or a stream like Kafka or Kinesis)? In fact you might even be able say every database already has this ;) (binlog, oplog, etc)

> not the whole db

If rows are immutable, what part of the db is left not being immutable? If rows were immutable, doesn't that imply that any existing "ranges, searches, indexes, etc" are immutable too?

If you going to the effort of making a decentralized database, why not also decompose all of these parts from one another... no reason the tables (logs), indexes, search, etc have to live in the same database, they could be spun off as completely different parts. Basically a something that indexes logs and then something else that takes those indexes and makes the searchable.

In fact this is all the rage right now with centralized databases... all of the work being done with streaming systems just seems to be an effort of decomposing and inverting databases... every old is new again.

Anyways, I agree with the idea and don't really know enough about decentralized systems to really understand why such a distributed database can't or hasn't already been built.

Post reply on HN