Around the World with SQLite3 and Rsync • Hacker News Board

Re: Around the World with SQLite3 and Rsync

#21

Post by sosodev » Fri, Jun 23, 2023, 5:34 PM UTC

tptacek wrote: post #17

sixhobbits wrote: post #9

This feels like a pretty significant step down in terms of quality for technical content output from Fly.io? To be fair, "explains a solution without explaining a problem" is one of my most common criticisms for technical content in general, but I am pretty sure this wouldn't have gotten to front page if it wasn't for fly's reputation. There's a lot of passive voice which makes it really hard to figure out who is doi…

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

It's strange that you would even ask. HN comments are notoriously critical and the standard for good content shifts extremely quickly around here. Why should Fly.io get a pass when nobody else does?

Re: Around the World with SQLite3 and Rsync

#22

Post by sixhobbits » Fri, Jun 23, 2023, 5:38 PM UTC

sixhobbits wrote: post #20

tptacek wrote: post #17

Earlier quoted context omitted.

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

I nearly deleted my comment because I felt bad after the author replied and also because I generally want to encourage people to publish stuff they are not confident about -- and I would never have posted this about a personal blog. But I decided to leave it because I think I stand by the implication that reputation for specific technical blogs can be earned or used up. Fly.io has earned a lot of reputation and I (an…

And just to be clear to your point of "not interesting enough" this is entirely about form, not the topic. If the content itself is interesting or not is hard for me to judge because of the obstacles presented by the form.

Re: Around the World with SQLite3 and Rsync

#23

Post by tptacek » Fri, Jun 23, 2023, 5:50 PM UTC

sosodev wrote: post #21

tptacek wrote: post #17

Earlier quoted context omitted.

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

It's strange that you would even ask. HN comments are notoriously critical and the standard for good content shifts extremely quickly around here. Why should Fly.io get a pass when nobody else does?

I guess I'm saying everybody should get a pass? I sort of don't even know what a "pass" means here. We should be judging every post here on its merits, and not holding them up to a weird, inchoate standard that both impedes us from writing straightforward useful posts, and also makes us reconsider the implications of posting deep dive technical stuff.

Re: Around the World with SQLite3 and Rsync

#24

Post by 0xbadcafebee » Fri, Jun 23, 2023, 5:52 PM UTC

Author, please rewrite this. My suggestions:

1) Delete the first two paragraphs.

2) Tell me what you're going to tell me, but without all the highlights and bullet points.

3) Sell me on why I want to read this article. Don't focus on the technical; focus on the human, what I as a person will get from it.

4) In later technical sections, don't give me instructions. Tell me what you are doing in general to solve some problem, then just show me the commands, and I will infer that I could also do X, do Y, do Z.

5) If you want to give a lot of technical detail and backstory, either put it in an expand, or in some kind of colored box so I have the visual context that I can probably skip this whole, continue the story, and come back to this later.

6) The recap shouldn't be larger than two paragraphs.

Re: Around the World with SQLite3 and Rsync

#25

Post by sosodev » Fri, Jun 23, 2023, 6:04 PM UTC

tptacek wrote: post #23

sosodev wrote: post #21

Earlier quoted context omitted.

It's strange that you would even ask. HN comments are notoriously critical and the standard for good content shifts extremely quickly around here. Why should Fly.io get a pass when nobody else does?

I guess I'm saying everybody should get a pass? I sort of don't even know what a "pass" means here. We should be judging every post here on its merits, and not holding them up to a weird, inchoate standard that both impedes us from writing straightforward useful posts, and also makes us reconsider the implications of posting deep dive technical stuff.

I see what you're saying. It just seems odd to say that fly.io is considering posting less because HN users are expecting too much or too critical. Surely HN isn't the only source of traffic to your blogs, right?

I love fly.io and appreciate the content but that doesn't sit right with me.

Re: Around the World with SQLite3 and Rsync

#26

Post by tptacek » Fri, Jun 23, 2023, 6:16 PM UTC

sosodev wrote: post #25

tptacek wrote: post #23

Earlier quoted context omitted.

I guess I'm saying everybody should get a pass? I sort of don't even know what a "pass" means here. We should be judging every post here on its merits, and not holding them up to a weird, inchoate standard that both impedes us from writing straightforward useful posts, and also makes us reconsider the implications of posting deep dive technical stuff.

I see what you're saying. It just seems odd to say that fly.io is considering posting less because HN users are expecting too much or too critical. Surely HN isn't the only source of traffic to your blogs, right? I love fly.io and appreciate the content but that doesn't sit right with me.

It's just an incoherent and unfair criticism. It's not Sam's job to write a deep dive on sqlite internals (that's Ben's job!). Sam's a Rails person, he writes about Rails stuff, as a search for [sam ruby rails] will indicate.

I just worry about us writing less stuff because people assume there's going to be some stunt driving sequence or shootout or CGI or whatever, and not everything people write here is like that.

Minute later

(I may be getting pushed into responding adversarially here just by message board dynamics. Sorry. Like I said up front: it's a weird thing to write! But what am I going to do, filter my thoughts before writing them on HN?)

Re: Around the World with SQLite3 and Rsync

#27

Post by samruby » Fri, Jun 23, 2023, 8:24 PM UTC

sixhobbits wrote: post #20

tptacek wrote: post #17

Earlier quoted context omitted.

This feels weird to say, but, honestly, and I feel sheepish saying this, when it comes to technology stuff I'm an HN person first and everything else second, so I guess I get to say it: Can I ask you not to say things like this? Specifically: this idea that there's a bar for technical stunt driving that we need to clear to get things up on the blog. I know I can't demand that of anybody; I'm just asking. I can't tell…

I nearly deleted my comment because I felt bad after the author replied and also because I generally want to encourage people to publish stuff they are not confident about -- and I would never have posted this about a personal blog. But I decided to leave it because I think I stand by the implication that reputation for specific technical blogs can be earned or used up. Fly.io has earned a lot of reputation and I (an…

> I felt bad

Don't. I can honestly say that I didn't write this post targeting HN. I'll go further... this post wasn't meant for people who are unlikely to use https://github.com/fly-apps/dockerfile-rails#overview. I recently added some features to that gem whose usage may not be intuitively obvious. I wrote this post to explain some of the motivation for those features.

I don't know how to mark posts as not intended for HN (and truth be told, if there was such a feature, I'd be inclined to overuse it). I don't know where else I should have posted this content, but I'm not sure I would be inclined to move it. In any case, this post, as written, serves a purpose for me. Somebody not you and not me felt it belonged here. We can both second guess that decision. Either way, there is no reason for either of us to feel bad.

Re: Around the World with SQLite3 and Rsync

#28

Post by pipo234 » Mon, Jun 26, 2023, 9:14 AM UTC

nateabele wrote: post #15

pipo234 wrote: post #7

Earlier quoted context omitted.

We sell live http media streaming and use sqlite3 locally to solve most of the hairy race conditions you would expect when serving files while they are simultaneously being written to. Despite our advice and common wisdom, every now and then customers complain about corrupted databases and after talks we discover they are using rsync or NFS to replicate the sqlite DB file. So, I was very interested in this article wh…

This is the blog post I want to read. What’s your setup for that?

So the absolute killer features that I would say SQLite offers are: 1. Embedded (single, small library, very fast) 2. Portable concurrency using native OS/FS abstractions. Full SQL support is nice, but I'd also be happy if it only offered plain key values. I love that you don't need to manage DB processes, network connections, FS partitions, shard memory, mutexes — just point your process or thread to a file and it magically fixes atomicity, consistency, transactions, the whole enchilada.

SQL databases are not great at storing huge chunks to data (petabytes of video) but those are easily handled by the native filesystem. Just make sure the write access is safely wrapped in a DB transaction, RAII style. This works and scales well enough except for the failure mode: over stressed egress (typically SELECT queries) will starve ingest (a more or less constant load of UPDATE & INSERT queries).

At that point, you want to scale horizontally. As mentioned, replicating the SQL backend file over NFS or rsync is simply a horrible idea. Surprisingly we ultimately discovered that SQL-level replication did not solve the problem reliably either: scale improved, but the starvation failure mode (read locks blocking DB writes) persisted (be it less frequently).

The replication layer we needed was at the application level where it's much easier to express that actually we did not need the real-time coupling between media ingest and egress. Eventual consistency is good enough: if the system is about to collapse, simply fail by adding latency so you can always keep ingest healthy.

Worst case, people watching a popular sports event will see players buffering. Which is bad — don't get me wrong — but far better than a catastrophically blocked pipeline and congested video encoders.