Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Around the World with SQLite3 and Rsync

fly.io

Post reply on HN
Print view View original

11–20 of 28 posts

Re: Around the World with SQLite3 and Rsync

#11

Post by Mizoguchi »

sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

Never heard of them and that was exactly what I thought. "What problem are they trying to solve?".

Re: Around the World with SQLite3 and Rsync

#12

Post by barnabask »

samruby wrote: post #10
sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

Author here, and fly.io employee. I have an app that allows dance studios to schedule "heats" for competitions. Multiple dance studios, in separate locations, use this app with separate databases. I merely blogged some techniques that I found useful for my app. Most apps won't need any of this, but in the future when somebody has a related problem, I wanted something I could point to to show some techniques that migh…

Why rsync and not Fly.io's own https://fly.io/docs/litefs/?

Re: Around the World with SQLite3 and Rsync

#13

Post by samruby »

barnabask wrote: post #12
samruby wrote: post #10

Earlier quoted context omitted.

Author here, and fly.io employee. I have an app that allows dance studios to schedule "heats" for competitions. Multiple dance studios, in separate locations, use this app with separate databases. I merely blogged some techniques that I found useful for my app. Most apps won't need any of this, but in the future when somebody has a related problem, I wanted something I could point to to show some techniques that migh…

Why rsync and not Fly.io's own https://fly.io/docs/litefs/ ?

The short answer is that litefs uses a single primary node (see: https://fly.io/docs/litefs/proxy/). I work with Ben so I imagine that over time we'll find a configuration that works with multiple databases each with their own primary node, but until then, this works for now.

Re: Around the World with SQLite3 and Rsync

#15

Post by nateabele »

pipo234 wrote: post #7
EGreg wrote: post #5

This can certainly work for files, if you don’t use a CDN. But also, you can just use a CDN! As for the database that handles your business logic … 3 minutes of googling would show that rsync is a poor (or at least a brittle) substitute for multi-master replication when it comes to database engines like InnoDB for MySQL (which is what most websites on the Internet use): https://dba.stackexchange.com/questions/91287/i…

We sell live http media streaming and use sqlite3 locally to solve most of the hairy race conditions you would expect when serving files while they are simultaneously being written to. Despite our advice and common wisdom, every now and then customers complain about corrupted databases and after talks we discover they are using rsync or NFS to replicate the sqlite DB file. So, I was very interested in this article wh…

This is the blog post I want to read. What’s your setup for that?

Re: Around the World with SQLite3 and Rsync

#16

Post by pstuart »

pipo234 wrote: post #7
EGreg wrote: post #5

This can certainly work for files, if you don’t use a CDN. But also, you can just use a CDN! As for the database that handles your business logic … 3 minutes of googling would show that rsync is a poor (or at least a brittle) substitute for multi-master replication when it comes to database engines like InnoDB for MySQL (which is what most websites on the Internet use): https://dba.stackexchange.com/questions/91287/i…

We sell live http media streaming and use sqlite3 locally to solve most of the hairy race conditions you would expect when serving files while they are simultaneously being written to. Despite our advice and common wisdom, every now and then customers complain about corrupted databases and after talks we discover they are using rsync or NFS to replicate the sqlite DB file. So, I was very interested in this article wh…

Use the backup api and expose that as a means of copying the database.

https://sqlite.org/backup.html

Re: Around the World with SQLite3 and Rsync

#17

Post by tptacek »

sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it:

Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking.

I can't tell you how many things we haven't written on our main blog because we're worried about whether it's as technically interesting as the "Gossip Glomers" thing we did with Aphyr.

Sam's a good writer and this is a pragmatic post about an approach he took to building something. The world is incrementally better with this post in it than without it. Holding off until we could have integrated LiteFS and written a 4 paragraph coda about sqlite3 internals would not make the world a better place.

I worry all the time about "Fly.io fatigue" on HN, for what it's worth! But the only sane response we can have to do that is: we just write what we want to write, and we let the community do what it's going to do. If 'dang gave us a header we could use to dampen some of our posts, we'd use it. :)

If I'm crazy to ask this, say so. We've been grappling with this for years now; if we knew exactly how to handle it, I wouldn't have to write weird things like this.

Re: Around the World with SQLite3 and Rsync

#18

Post by ignoramous »

tptacek wrote: post #17
sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

In fairness (and like OP mentioned), you folks set the bar really high yourselves.

> But the only sane response we can have to do that is: we just write what we want to write, and we let the community do what it's going to do.

Move over paxos and sqlite3; HN needs FlyLLM and Fly vGPU to satiate its appetite now ;)

Re: Around the World with SQLite3 and Rsync

#19

Post by tptacek »

ignoramous wrote: post #18
tptacek wrote: post #17

Earlier quoted context omitted.

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

In fairness (and like OP mentioned), you folks set the bar really high yourselves. > But the only sane response we can have to do that is: we just write what we want to write, and we let the community do what it's going to do. Move over paxos and sqlite3; HN needs FlyLLM and Fly vGPU to satiate its appetite now ;)

GPUs are coming! I lost a big chunk of 2023 to GPU driver stuff.

Re: Around the World with SQLite3 and Rsync

#20

Post by sixhobbits »

tptacek wrote: post #17
sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

I nearly deleted my comment because I felt bad after the author replied and also because I generally want to encourage people to publish stuff they are not confident about -- and I would never have posted this about a personal blog.

But I decided to leave it because I think I stand by the implication that reputation for specific technical blogs can be earned or used up. Fly.io has earned a lot of reputation and I (and I guess others) clicked on a title I would not usually have because I saw the domain. In this case I feel like some reputation was "used up" and I am (slightly) less likely to click on a post solely because I see a fly.io domain in future.

Maybe I am just not the target audience for the post and it is better than my assessment, but if not then I think people value HN as an honest audience that doesn't hold back on criticism (I know I do and I have received harsh but valid criticism from hn on stuff I have written).

Fly.io can probably keep making front page based on reputation if there is a balance of very high quality content and other posts, but I feel like something like this could have been 1000% better with another revision round and some editing (probably only 20% more effort) so I will keep saying stuff like this I guess?

Post reply on HN