Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LiteFS

fly.io

Post reply on HN
Print view View original

151–158 of 158 posts

Re: LiteFS

#151

Post by yencabulator »

endisneigh wrote: post #83

Seems neat, until you try to do schema migrations. Unless they can guarantee that all containers’ SQLite instances have the same scheme without locking I’m not sure how doesn’t run into the same issues as many NoSQL. CouchDB had this same issue with its database per user model and eventually consistent writes.

I don't see how this is different to a world where you use Postgres over the network, with lots of DB clients. You "just" do the schema migrations like everyone who can't shut down the business does: incrementally and in a backwards-compatible manner. After the migration, deploy new code that takes advantage of the new schema. Repeat until done.

Re: LiteFS

#152

Post by yencabulator »

presentation wrote: post #140

My dream would be if this supported geo-partitioning. In my field people are pretty sensitive about GDPR so would love to box in EU PII in EU servers.

Since the scope is a SQLite database, that means you just run a EU database, with no replicas outside of EU.

This level of database replication isn't going to put individual rows in different geographies.

Re: LiteFS

#153

Post by ngrilly »

yencabulator wrote: post #147
ngrilly wrote: post #138

Earlier quoted context omitted.

Can the app read it concurrently while it is updated by LiteFS?

Looks like LiteFS invalidates the cached file contents on every update of position. This means there is never stale data to be read. (The client needs to make sure to consume that in a single pread/read syscall, or it could observe a sheared state.) Disclaimer: I wrote the FUSE framework LiteFS uses, https://bazil.org/fuse https://github.com/superfly/litefs/blob/52e269d4b04070690ce2... https://github.com/superfly/lit…

Very clear. Thanks for the great explanation.

Re: LiteFS

#154

Post by pweissbrod »

simonw wrote: post #80
pweissbrod wrote: post #67

Earlier quoted context omitted.

Select N+1 is a fundamental anti-pattern of relational database usage. Reducing the latency per round trip doesn't change this fact.

I disagree. I think it makes a big difference. https://sqlite.org/np1queryprob.html#the_need_for_over_200_s... talks about this in the context of Fossil, which uses hundreds of queries per page and loads extremely fast. It turns out the N+1 thing really is only an anti-pattern if you're dealing with significant overhead per query. If you don't need to worry about that you can write code that's much easier to write an…

Perhaps the overhead of chatty queries is diminished in this special use case.

But it doesn't change the fact that standalone database server processes are designed to support specific queries at lower frequencies. This is one of the main points of The SQL language is to load precisely the data that is needed in a single statement.

Relying on this is a design pattern would only scale in specific use cases and would hit hard walls in changing scenarios

Re: LiteFS

#155

Post by cbsmith »

yencabulator wrote: post #144
cbsmith wrote: post #105

Earlier quoted context omitted.

> So your read queries should mostly be measured in microseconds. You should check out the read latency for read-only requests over unix domain sockets with PostgreSQL. You tend to measure it in microseconds, and depending on circumstances it can be single-digit microseconds. Regardless of whether your FUSE logic does nothing at all, It sure seems like there's intrinsic overhead to the FUSE model that is very similar…

Yes, FUSE adds overhead. However, 1. kernel page cache fully removes the read overhead for in-memory pages 2. There's a FUSE_PASSTHROUGH mode that removes that overhead for all reads & writes. I haven't studied exactly what writes LiteFS needs to observe (just journal vs all data writes), but at least for reads it seems to pass them straight through. We could well submit a FUSE_PASSTHROUGH_READ patch to the kernel, a…

I had totally forgotten about FUSE_PASSTHROUGH. That's a great insight. Thanks.

Re: LiteFS

#156

Post by tptacek »

ignoramous wrote: post #148
tptacek wrote: post #123

Earlier quoted context omitted.

No? No. No!

You won't change my mind with 3 No 's. A fourth however...

I had literally nothing to do with Fly.io's switch from Javascript to Firecracker. Happened months and months before I joined.

Re: LiteFS

#157

Post by ignoramous »

tptacek wrote: post #156
ignoramous wrote: post #148

Earlier quoted context omitted.

You won't change my mind with 3 No 's. A fourth however...

I had literally nothing to do with Fly.io's switch from Javascript to Firecracker. Happened months and months before I joined.

I don't want to come off as downplaying others engs at Fly (esp those from Compose.io) [0], but I wanted to point out how there's a marked difference between the period from when you joined them and years preceding that. A catalyst?

Of course, I claim no inside knowledge, so I may very well be mistaken (:

[0] https://news.ycombinator.com/item?id=9937023

Re: LiteFS

#158

Post by tptacek »

ignoramous wrote: post #145
stingraycharles wrote: post #68

Earlier quoted context omitted.

What is @tptacek’s relation with fly.io, and how did he change it?

tptacek is a security researcher and principal at fly.io [0]. And as an outsider looking in, seems to have been the eng behind some of their defining features [1]: - udp support - container2vm overhaul - private networks aka 6pn - some key flyctl (cli) commands like flyctl ssh, flyctl proxy - metrics - litefs - perhaps, the imminent overhaul of the orchestration layer (?) - the upcoming authz layer [0] https://archiv…

GAH NO.
Post reply on HN