Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky migrates to single-tenant SQLite

github.com

Post reply on HN
Print view View original

221–230 of 247 posts

Re: Bluesky migrates to single-tenant SQLite

#221

Post by PurpleRamen »

jakebsky wrote: post #219
PurpleRamen wrote: post #211

Earlier quoted context omitted.

This looks like someone is building their own filebased database-system, in typescript, while still using mature features of database-servers. So instead of trusting the optimized, regularly maintained and battletested solution, they build something by themselves. This smells ugly, like something that will scale poor in performance, and will have security and tooling-problems. Simplification of installation seems not…

Using SQLite is most certainly not "building their own filebased database-system" SQLite is just about as mature and well-tested as it gets in the entire world of software: https://www.sqlite.org/testing.html Each users' data is naturally partitioned at the atproto repository level, so this is the sweet spot for per-user SQLite databases. It would make total sense for a PDS instance to have just a single user on it,…

> Using SQLite is most certainly not "building their own filebased database-system"

[..] Each user has their own SQLite file [..]

[..] We also introduce 3 separate SQLite databases for managing service state [..]

This doesn't use SQLite for the database-managment, but for the individual "document". The database-managment itself is handled in the application-server. You jiggle around files and poke wherever it matches, this is by a classical filebased database-managment-system.

> It would make total sense for a PDS instance to have a single user, and in fact that is likely for many self-hosters.

Sure, if it's just a low-user-instance, the performance is not much of a deal. But from my impression here, this is also the code Bluesky uses for everything else, from low to massively high user-instances. And then I want to see how RAM holds up, when you have 10k+ user-databases open at the same time on one instance.

> There are trade-offs to this kind of a system but it comes out way ahead in this case.

Which is why I want to see some actual numbers and solid explanations going more into details then the gossip in the comments here.

> A major consideration is that we're planning to run at least 100+ instances, which would require operating 100+ high availability (primary+replica) Postgres clusters.

Are those independent instances, or just 100+ instances servers from the same company on different locations? But I don't see how this can replace a whole postgres-cluster without removing significant functionality. I mean sqlite does not have good replication on it's own AFAIK, so as you seem to still use replication, you just replace it with another solution? Which also means you remove the same options for anyone else, and forces them to use your solutions?. I don't see how this will be beneficial for self-hosters.

Re: Bluesky migrates to single-tenant SQLite

#222

Post by dathinab »

MenhirMike wrote: post #45
lrx wrote: post #29

Earlier quoted context omitted.

I prefer to read the unified diff and commits don't matter as much.

Same. Do whatever you want in your feature branch, what matters is the Files list and the description in the PR. The whole thing gets squashed into a single commit anyway (which also makes reverting much easier).

I my experence with teams as long as you

1. require reviews before merging

2. have not very disjunct PRs (sometimes for e.g. legacy maintenance projects you mostly have disjunct PRs normaly you do not)

then you need stacked PRs for productivity, i.e. you need to be able to continue working on a new PR based on the old PR before that is fully merged (or reviwed).

In this case in my experience three workflow work:

1. you (may) squash commits, and rebase stacked PRs once the previous PR has been merged (or sometimes majorly modified, but that quite advanced rebase usage). This works but has some major pain points: 1) rebased during reviews are terrible bad handled by github, 2) git doesn't keep track of the original start of a branch, this can lead to issues if you squash the commits when merging, 3) no good build in tooling for it

2. All forms of history manipulation are forbidden including rebasing and squashing. It's merge only because of this git doesn't get confused when merging squashed commits and everything seems fine... Until you now realize that follow up changes from reviews of a parent PR happen in the git history chronologically after your follow up PR and that can be a total pain depending on what changes. (Through you are allowed to fully rebase your history before marking a PR as ready for review so as long as the "stack" of PRs isn't too deep it's fine).

3. you agree with Linus that github PRs have major issues and go with a patch based approach for merging, now you need completely different tooling which often less nice modern UI but id doesn't have any of the issues of point 1 or 2

It's was quite a wtf are you doing industry moment when I realized that the most widely used contributions flows (weather in open source or in companies) are either quite flawed (1&2), productivity nightmares (no stacked commits) or quite inconvenient (3).

Re: Bluesky migrates to single-tenant SQLite

#223

Post by dathinab »

readline_prompt wrote: post #34
lrx wrote: post #29

Earlier quoted context omitted.

I prefer to read the unified diff and commits don't matter as much.

don't know why, but recent teams around me have always made strict rules about number of commits in PRs. I just wanted to tell them the same thing you said: "Why don't you just look at the diffs?" curious for other opinions. (sorry not really about this particular topic)

Maybe it's just an approach to try to force logically smaller PRs without trying to limit the number of lines changes.

I.e. with an idea like:

- if we try to commit so that each commit does a singular change

- then by limiting the number of commits we limit the number of "logical" changes in a PR

- and in turn make reviews and similar easier

Re: Bluesky migrates to single-tenant SQLite

#224

Post by gigatexal »

emadda wrote: post #218
gigatexal wrote: post #145

At a previous fintech role the company would store customer accounts as encrypted sqlite3 files on blob storage ... this worked out decently well for our access patterns.

How did they lock the file when re-uploading it after edits?

Each was encrypted before and after. The keys were stored securely and retrieved temporarily and then rotated I think. It’s a process I wasn’t 100% privy too as I was too junior.

Re: Bluesky migrates to single-tenant SQLite

#225

Post by _heimdall »

rchaud wrote: post #200
_heimdall wrote: post #180

Earlier quoted context omitted.

Are you assuming that hate and anti-intellectualism are exclusively a rightwing thing?

On Twitter, the place that hired Tucker Carlson after Fox News dumped him? Yeah it is. No need for "both sides"-ing on this one.

One example is hardly proof of an absolute though. Assuming all conspiracy or anti-intellectual thought comes from one side because Tucker Carlson is a huge logical misstep in my opinion

Re: Bluesky migrates to single-tenant SQLite

#226

Post by krainboltgreene »

lolinder wrote: post #101
krainboltgreene wrote: post #61

Earlier quoted context omitted.

Funny that two of your commits don't actually tell us why they exist, one simply describes the diff (which you should never need lol?) and the other proxies that responsibility to some other system. You could have simply randomized the text in each commit, put the ticket id and the one "why" in the merge commit body and gotten the same end result amount of real information in the end.

The first line of the commit message isn't about including information that couldn't be gleaned from the commit. That can be done in subsequent lines. The first line is for two purposes: * Priming the reader so they are able to quickly interpret what they're seeing when they open the commit. * Making it easy to search or scan for a specific change. The last commit message in my example would probably have included th…

> Making it easy to search or scan for a specific change.

I'm trying to imagine the near infinite terms I would have to search for to find the commit where I "changed from a hash to a set".

Regardless, every other thing you said could also just be done in the central PR body (and thus the merge commit) and be much easier to access.

Instead of "priming the reader" it's infinitely more helpful to tell the reader why you did something, because you can't extract that from a diff.

Re: Bluesky migrates to single-tenant SQLite

#227

Post by whyrusleeping »

winrid wrote: post #191
moreati wrote: post #122

Earlier quoted context omitted.

At a guess: that hash is performed relatively few times, so any performance difference is lost in the noise floor. Never having to answer "why did you use this insecure hash" or eliminating/minimising any possibility of a class of security problem is worth more.

This has nothing to do with security. It's just wasted CPU. I imagine you have to do this every time you make a query to lookup the users DB? Security is not a concern here. It's just literally bucketing ids. Also, this is not needed with modern file systems.

all modern server CPUs have intrinsics for sha256, it just doesnt matter CPU-wise

Re: Bluesky migrates to single-tenant SQLite

#228

Post by defatigable »

pests wrote: post #105
defatigable wrote: post #55

Earlier quoted context omitted.

I have some extra if you'd like one. Let me know how to get it to you and I will.

I'm still trying to get one if anyone see's this. Keep missing the ones posted. Email in profile. Thanks.

emailed you with one
Post reply on HN