Live data from Hacker News

Database of Databases

dbdb.io

61–65 of 65 posts

Re: Database of Databases

#61
post #34

Earlier quoted context omitted.

SQLite isn't single threaded - and it can support multiple readers very well. The limitation with SQLite is that it doesn't support concurrent writes well - it needs to take a lock on the entire database to perform a write. Writes are crazy fast (a few ms) so this often isn't a problem - but it does mean you wouldn't want to use it to build a site that has many people writing at once, like Hacker News for example. Fo…

Enabling the write ahead log makes sqlite behaviour much, much better under (write) contention: > PRAGMA journal_mode=WAL;

Yeah I was getting occasional "database is locked" read errors on a project that had crons writing to the SQLite file which I solved by switching on WAL mode.

It still doesn't let you have concurrent writes but it does mean that reads won't error if a write is going on at the same time.

Re: Database of Databases

#64

Thought it was some groundbreaking new database technology.... Speaking of, What would an interesting version of "database of databases" look like?

This is basically what we're building at Splitgraph [0]. We're calling it a "data delivery network." You connect to one SQL endpoint and can query (and join across) 40k+ different datasets. It's built on Postgres, and as far as your SQL client is concerned, it's talking to a Postgres database with 40k tables in it. Right now we forward queries to public data portals, but eventually you'll be able to connect live data sources to the DDN without writing any code. We want it to be as easy as configuring Cloudflare; you just upload a set of read-only credentials in the web UI and we take care of the rest. For more private use cases, we're planning to offer private deployments to AWS/GCP/Azure.

Technically, this is database virtualization, which isn't really a new concept. We're implementing it as a database proxy, using PgBouncer instances to intercept queries and route them to Splitgraph engines. Within a Splitgraph engine (which is Postgres + some custom code), each "table" is either a "mounted" live database via a foreign database wrapper (FDW), or part of a point-in-time, versioned database snapshot called a "data image" that you can build with sgr.

[0] https://www.splitgraph.com

Post reply on HN