Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky migrates to single-tenant SQLite

github.com

Post reply on HN
Print view View original

21–30 of 247 posts

Re: Bluesky migrates to single-tenant SQLite

#21

Post by starcraft2wol »

lucasyvas wrote: post #19

Love SQLite - in general there are many challenges with a schema or database per tenant setup of any kind though. Consider the luxury of row-level security in a shared instance where your migration either works or rolls back. Not now! If you are doing a data migration and failed to account for some unexpected data, now you have people on different schema versions until you figure it out. Now, yes, if you are at shard…

> now you have people on different schema versions until you figure it out.

That can be a good thing if your product has say I guess it depends on the business structure.

Re: Bluesky migrates to single-tenant SQLite

#22

Post by nvy »

lucb1e wrote: post #18
nvy wrote: post #13

Earlier quoted context omitted.

Yes. I have invite codes if you would like one. Email in my profile. Edit: they're all gone!

Was browsing around your website (mentioned in profile), noticed https://0x85.org/contact.html only mentions Twitter and email. Maybe the bluesky omission is intentional, but probably it just hasn't been updated yet? I'm not on bsky myself, currently having fun on mastodon and I'm not familiar with bsky enough to know what I'm missing out on, but for other folks I figured I'd mention it

Hey thanks. It's just outdated, what with young kids and grad school. Appreciate the note.

Re: Bluesky migrates to single-tenant SQLite

#23

Post by starcraft2wol »

sinkwool wrote: post #20
malkia wrote: post #16

What do they mean by "Since SQLite does not support concurrent transactions" - it supports them, as long as you don't access the .db file through a file share (UNC, or NFS, etc) - https://www.sqlite.org/wal.html I've been using this to update/read db from multiple threads/processes on the same machine. You can also do snapshotting with the sqlite backup API, if you want consistent view, and to not hold on transaction…

> Writers merely append new content to the end of the WAL file. Because writers do nothing that would interfere with the actions of readers, writers and readers can run at the same time. However, since there is only one WAL file, there can only be one writer at a time. I think the OP meant that updates have to run sequentially.

Which just means the lock happens at user scope in this case instead of per table or row. This limitation still causes so much confusion when it’s a completely reasonable design.

Re: Bluesky migrates to single-tenant SQLite

#24

Post by jakebsky »

Retr0id wrote: post #17
wmf wrote: post #6

Earlier quoted context omitted.

https://blueskyweb.xyz/blog/5-5-2023-federation-architecture

To summarise the relevant details, the "AppView" service is responsible for the sorts of queries that aggregate across users, and that has its own database setup - I think postgres but I'm not 100% sure on that.

You're right, as usual. AppView is on a Postgres cluster with read replicas doing timeline generation (and other things) on-demand. We're in the process of moving it toward a beefy ScyllaDB cluster designed around a fanout-on-write system.

The v1 backend system was optimized for rapid development and served us well. The v2 backend will be somewhat less flexible (no joins!) but is designed for much higher scale.

Re: Bluesky migrates to single-tenant SQLite

#25

Post by lucasyvas »

starcraft2wol wrote: post #21
lucasyvas wrote: post #19

Love SQLite - in general there are many challenges with a schema or database per tenant setup of any kind though. Consider the luxury of row-level security in a shared instance where your migration either works or rolls back. Not now! If you are doing a data migration and failed to account for some unexpected data, now you have people on different schema versions until you figure it out. Now, yes, if you are at shard…

> now you have people on different schema versions until you figure it out. That can be a good thing if your product has say I guess it depends on the business structure.

Totally correct. But not a good thing in our case!

Re: Bluesky migrates to single-tenant SQLite

#28

Post by numbsafari »

scorpion_farmer wrote: post #27

That looks like the PR from hell - 190 files changed, 143 commits? Mostly with names like "tidy" and "wip" Props to whoever actually reviewed that, you are a warrior

> Props to whoever actually reviewed that, you are a warrior

Or a ghost.

Post reply on HN