Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Distributed SQLite: Paradigm shift or hype?

kerkour.com

Post reply on HN
Print view View original

51–60 of 165 posts

Re: Distributed SQLite: Paradigm shift or hype?

#51

Post by rozenmd »

_heimdall wrote: post #44
skybrian wrote: post #41

I don't believe Cloudflare D1 has read replicas yet? They describe how it will work in detail in a blog post [1], but in the future tense. > We’re actively working on global read replication and realizing the above proposal (share feedback In the #d1 channel on our Developer Discord). Perhaps it will be out by the time the book is finished. [1] https://blog.cloudflare.com/building-d1-a-global-database

D1 is still pretty limited in my experience. No read replicas really kills any of the meaningful benefit of the architecture, and being built off of a (mostly) SQLite-compliant API makes me nervous as it isn't really SQLite at all. Last year's major Cloudflare outage really was the final straw for me with regards to D1. I don't mean that as a knock at Cloudflare at all, the situation sounded horrible and I appreciate…

D1 (the ability to query your database at the edge) didn't get knocked offline during that outage though?

Re: Distributed SQLite: Paradigm shift or hype?

#53

Post by LVB »

danappelxx wrote: post #12
zackify wrote: post #8

I think this skips one mega benefit for apps. I’ve been using liteFS in production for a couple months. Your web app is able to resolve db queries instantly. You don’t need loading states if you’re using complex charts and other frontend JS that waits for data. All the data is resolved so fast and you can just return all your data like more traditional apps, and the load times are insane. If you’re multi region you c…

I think you always need loading states to account for slow network, or am I missing something?

>to account for slow network

Or a slow anything else, e.g., SQLite queries. This thread has focused on the network aspects, and they do stand out since there can be such a large gap between a network call vs a local SSD read. But we're still talking about a database, which could be huge (presumably it's the main, single DB for the whole app). And there is still all of the actual SQLite work that needs to happen to execute a query, plus opportunities for really bad performance b/c of bad queries, lack of indexes, all the usual suspects. Not to mention the load on the owning process itself. So, I'm agreeing that a loading state is needed.

Re: Distributed SQLite: Paradigm shift or hype?

#54

Post by tptacek »

internetter wrote: post #52

Something I've been thinking about is partitioning my SQLite. Instead of storing all user's data in one mega table, what if I made a SQLite database for each user? Provided users never talk to each other, I think this might work?

You can attach to databases dynamically in queries and join across them. I probably wouldn't (in an ordinary data model) do per-user, but I would consider it for different functional areas.

Re: Distributed SQLite: Paradigm shift or hype?

#55

Post by znkr »

galaxyLogic wrote: post #13

What portion of writes typically have to go to the primary server? If users are local then their data maybe is often local as well. In other words not all writes need to first go the central server from where they are distributed back to all edge-databases. I can see that the organization running the application needs a global view of all data. But regional users perhaps don't. Often they just need to know their own…

It’s been done and it becomes quite complicated very quickly. Then it becomes necessary to manage that complexity and you end up with something like Spanner.

Re: Distributed SQLite: Paradigm shift or hype?

#56

Post by danr4 »

internetter wrote: post #52

Something I've been thinking about is partitioning my SQLite. Instead of storing all user's data in one mega table, what if I made a SQLite database for each user? Provided users never talk to each other, I think this might work?

I believe this is what one of the companies mentioned in the article, Turso, can help you do. a per-tenant database.

Re: Distributed SQLite: Paradigm shift or hype?

#57

Post by rapnie »

simonw wrote: post #27
danappelxx wrote: post #21

Earlier quoted context omitted.

Yup, exactly. Phones change wifi networks, routers drop packets, load balancers get overloaded. Hard to fully eliminate tail latencies.

The key here is to make a single API call to the backend which then runs 100+ SQL queries at once and combines the results into a single JSON response - that way you're only paying the network cost once. See https://www.sqlite.org/np1queryprob.html I've implemented GraphQL on top of SQLite and found it to be an amazingly good match, because the biggest weakness of GraphQL is that it makes it easy to accidentally trig…

> I've implemented GraphQL on top of SQLite and found it to be an amazingly good match

Could you give a pointer to the repository, or is this part of Datasette?

Re: Distributed SQLite: Paradigm shift or hype?

#58

Post by mind-blight »

xray2 wrote: post #24

Most apps never need to scale. Also, worried about scaling? Just use an ORM that allows you to switch from SQLite to Postgres. It’s as simple as that.

Unless your using database specific features. One of the biggest advantages for Postgres is how incredible the ecosystem is. It doesn't work for everything, but I have an OEM, multiple kinds of text search (vector, inverted indexes, trigrams), recursive and graph-like queries (though that's admittedly less of an issue if n+1 isn't a problem), row-level acls, locks, etc.

It's really nice to have all of that power available in one piece of infrastructure.

Re: Distributed SQLite: Paradigm shift or hype?

#60

Post by sorenbs »

internetter wrote: post #52

Something I've been thinking about is partitioning my SQLite. Instead of storing all user's data in one mega table, what if I made a SQLite database for each user? Provided users never talk to each other, I think this might work?

This setup can work great and even support elements that are shared between users, if you also give those elements their own DB. I’m working on a prototype to support this natively in Prisma.
Post reply on HN