Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky migrates to single-tenant SQLite

github.com

Post reply on HN
Print view View original

211–220 of 247 posts

Re: Bluesky migrates to single-tenant SQLite

#211

Post by PurpleRamen »

barnabee wrote: post #188
PurpleRamen wrote: post #185

On the surface, this looks like the worst combined with the awful. I hope someone will make a good article with some hard numbers to explain the benefits and analyze the assumed flaws, because this could be something really, fascinating to learn about.

Can you explain why this looks like the "worst combined with the awful" to you? To me, on the surface, particularly assuming you are building a distributed system to be run and deployed by many users, some of which are not professional sysadmins (which I believe is likely to be a goal here, and should be), this seems like quite a sane choice. I'd definitely expect a design goal to be avoiding the need to setup/config…

This looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems.

Simplification of installation seems not like a good enough reason to trust your whole backend on this. Installing and maintaining a database-server is not that hard today. This is well established and documented, unlike this. But I also don't know enough about this app, maybe this is just one of several options, meant for a specific usecase? Using this in a standalone desktop-app would make sense, while still offering a mature sql-backend available for server-installations.

Re: Bluesky migrates to single-tenant SQLite

#213

Post by pests »

numpad0 wrote: post #199
WinstonSmith84 wrote: post #144

Earlier quoted context omitted.

can't believe it's gone, too many smart people on this website lol

Not called hn for nothing! Glad I'm absolute bottomest on the floor in terms of intelligence or ability here

Nope that might be me. I started this comment thread and I didn't even manage to snag one despite getting direct replies multiple times with codes.

Re: Bluesky migrates to single-tenant SQLite

#214

Post by muttled »

PurpleRamen wrote: post #211
barnabee wrote: post #188

Earlier quoted context omitted.

Can you explain why this looks like the "worst combined with the awful" to you? To me, on the surface, particularly assuming you are building a distributed system to be run and deployed by many users, some of which are not professional sysadmins (which I believe is likely to be a goal here, and should be), this seems like quite a sane choice. I'd definitely expect a design goal to be avoiding the need to setup/config…

This looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems. Simplification of installation seems not…

I'm not a professional coder, only side-projects. Never formally taught. I looked at the solution and thought it kinda sounds like something I'd come up with. Like when I didn't know how to use data tables and would hold data in an array of arrays to form the rows and columns. Somewhat clever, "works", but would probably make my professional coder friends vomit if I explained it to them.

Re: Bluesky migrates to single-tenant SQLite

#215

Post by muttled »

alex_suzuki wrote: post #83
winrid wrote: post #63

Why sha256 hash the user into to get a two character target directory? Wouldn't md5 be much faster and solve the same problem?

This is probably not about collisions but about filesystem limitations (max number of files in a directory).

I've done something similar and that's absolutely what it was. I'm no pro, knew I wasn't doing it the right way, but it was for a personal side project and Windows starts to get weird when you have a million files in a single directory.

Re: Bluesky migrates to single-tenant SQLite

#216

Post by andrewstuart »

dathinab wrote: post #169
andrewstuart wrote: post #114

I sure hope they don’t ever want to change their db structure. Why not use Postgres with RBAC (Row Based Access Control).

- simpler db client - simpler cloud architecture - simpler resource management - simpler partial backups/restore - simpler compliance with law enforcement - partitioning might be easier, e.g. when handing "user account storage which should be undo-able for a while" (e.g. long term absent users data could be moved to cold storage, blocked/deleted users data could move to some scheduled for deletion space allowing undo…

You didn’t systemically document “harder”.

Re: Bluesky migrates to single-tenant SQLite

#219

Post by jakebsky »

PurpleRamen wrote: post #211
barnabee wrote: post #188

Earlier quoted context omitted.

Can you explain why this looks like the "worst combined with the awful" to you? To me, on the surface, particularly assuming you are building a distributed system to be run and deployed by many users, some of which are not professional sysadmins (which I believe is likely to be a goal here, and should be), this seems like quite a sane choice. I'd definitely expect a design goal to be avoiding the need to setup/config…

This looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems. Simplification of installation seems not…

Using SQLite is most certainly not "building their own filebased database-system"

SQLite is just about as mature and well-tested as it gets in the entire world of software: https://www.sqlite.org/testing.html

Each users' data is naturally partitioned at the atproto repository level, so this is the sweet spot for per-user SQLite databases. It would make total sense for a PDS instance to have just a single user on it, and in fact that is likely for many self-hosters. It's also worth noting that the PDS software already had SQLite support, which made this change somewhat easier.

There are legitimte trade-offs to this kind of a system but it comes out way ahead in this case, and it's not as wild as it may seem to those not familiar with the power of SQLite.

A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters. This would be a huge amount of operational and financial overhead.

We chose this route because it is better for us as a small team, with relatively limited resources. But it also has the property of being much easier for self-hosters, of which we hope there will be many.

Re: Bluesky migrates to single-tenant SQLite

#220

Post by wmf »

rapnie wrote: post #124
wmf wrote: post #6

Earlier quoted context omitted.

https://blueskyweb.xyz/blog/5-5-2023-federation-architecture

> The BGS handles "big-world" networking. It crawls the network, gathering as much data as it can, and outputs it in one big stream for other services to use. It’s analogous to a firehose provider or a super-powered relay node. "Big-world" networking by Big Tech-to-be Bluesky with super-powers, I wonder? Is this BGS also going to be federated, or is that the big centralized beating heart of this platform managed excl…

There can be multiple BGSes (like there are a few Web search engines) but it's expensive to run so there probably won't be many. Alternative designs are either more expensive or don't have the same features.
Post reply on HN