Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A database for 2022

tailscale.com

Post reply on HN
Print view View original

71–80 of 336 posts

Re: A database for 2022

#71

Post by dave78 »

Aeolun wrote: post #44
ramenmeal wrote: post #18

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they…

> why would an engineer want to join a company that is making these decisions? Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.

I would actually love to work at a company like that. I've been a professional developer for over 20 years now and at least in my experience, too much unneeded complexity has been the root cause of so many defects and product failures. I'm constantly fighting battles to keep things simple. All the young devs on my team want to use every fancy new technology that passes by, but I just want things to work for my customers with the smallest amount of downtime and late night pages.

If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.

Re: A database for 2022

#72

Post by acrispino »

jjeaff wrote: post #39
norom wrote: post #23

I for one am enjoying this. Sure they could probably just use a regular ol' database, but where is the excitement in that!

I really hope they aren't taking the same tack when dealing with the encryption portion of their service.

They're using WireGuard for the tunneling. I don't think a lot of people will object

Re: A database for 2022

#73

Post by cogman10 »

dan-robertson wrote: post #56
musjleman wrote: post #16

So to recap their timeline: * Instead of an actual database use a JSON file. * Write a blog post about how that didn't scale. * Instead of an actual database hand-roll something else. * Write a blog post about how that didn't scale. * Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it repl…

Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments. And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?

No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!"

> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.

Seriously.. consider this.. they are worried about the transaction semantics of using db replicas and instead CHOSE SQL LITE!

Re: A database for 2022

#74

Post by tptacek »

cogman10 wrote: post #65
dgb23 wrote: post #52

Earlier quoted context omitted.

Have a look at this headline: > Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere. I don't know how their product works, but my intuition is that your suggestions aren't feasible.

I suggest reading the article. The reason they haven't chosen "boring" tech has nothing to do with the clients running their software and everything to do with their internal development experiences. > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have…

If you're dinging them for not using Cockroach, they explained why they're not using Cockroach in the exact paragraph you quoted: if they use Cockroach, they're committing to the scaling and distribution system that Cockroach provides, which might or might not be a good fit for them 2 years from now. If it isn't a good fit, they're stuck with a huge engineering bill to get themselves out of Cockroach.

We're moving towards sqlite for things for the same reason. sqlite is simple to reason about. The distributed state problem not simple, but our distributed state problem is not Tailscale's and theirs isn't yours; every distributed state problem is unhappy in its own way. As they work out the contours of their specific distributed state problem, sqlite isn't the component that's going to break down.

Re: A database for 2022

#75

Post by tptacek »

cogman10 wrote: post #73
dan-robertson wrote: post #56

Earlier quoted context omitted.

Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments. And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?

No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!" > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is…

And?

(It's sqlite, or SQLite if you're being pedantic; it's not "SQL LITE").

Re: A database for 2022

#76

Post by rob_c »

lostcolony wrote: post #51
rob_c wrote: post #49

Earlier quoted context omitted.

Yes but userland wireguard is unique and useful. This is ignoring textbooks, competitors who write about their exploits warts and all and screams of get a more experienced db team to work with you... shoving it into a json screams of throw it at a disk and let the hardware deal with it

>> shoving it into a json screams of throw it at a disk and let the hardware deal with it Which...worked. For quite a long time. And required no dev effort. Freeing devs up to focus on other things. They did it as a POC, and it worked, better even than they had anticipated; why over-engineer it up front? Even when it turns out they are gonna need it (i.e., YAGNI no longer applies), they have been able to push off the…

Speaking as someone who has had to fix things like this...

When you reach capacity and it hits a screaming halt as the hardware capacity has been reached and performance doesn't collapse its completely gone.

This is not about best use. Hell json file performance may be well within spec of the system requirements, but jumping up and down without perf numbers or specs is typical of sweep it under the carpet for a generation...

Re: A database for 2022

#77

Post by benbjohnson »

chubot wrote: post #38

Honest question: what do the sqlite authors think of litestream? I seem to recall Richard Hipp on the "changelog" podcast mentioning it, but I don't remember what he said. 2 episodes here: https://changelog.com/person/drh/podcasts#feed I think it was very neutral, something to the effect of "there are multiple solutions". But I know essentially nothing about sqlite internals so I can't judge, or maybe that's why I di…

> Honest question: what do the sqlite authors think of litestream?

Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff that's what makes the new read replication work. They ended the call by asking me if there's anything they could help with or anything I needed.

> I guess I'm wondering if it's a recommended/supported mode of operation in sqlite.

I don't think the SQLite team endorses any tooling outside of what they build AFAIK. So in that sense, no, it's not officially supported. However, the API for maintaining checkpointing is publicly available and there are docs for controlling it from outside processes (which is what Litestream does).

> What are the failure cases? How much data can you lose?

Litestream is designed so that it keeps retrying in the event that you can't connect to a replication destination. S3 is pretty reliable so it may be a network outage that could cut you off. You can also enable Prometheus metrics to be reported out of Litestream if you want to add monitoring and alerting.

As far as data loss, by default it's setup to bundle database changes together every 1 second, compress them, and upload them to somewhere like S3. So your window for data loss is 1 second unless S3 goes down.

You can also run regular backups with the SQLite CLI and upload those as a fallback. That's what I typically do since it's really cheap and easy to setup an hourly cronjob and I'm overly paranoid. :)

Re: A database for 2022

#78

Post by cogman10 »

mrkurt wrote: post #66
cogman10 wrote: post #54

Earlier quoted context omitted.

> Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? It depends on what that tool is doing and if the thing it's doing is a core competency for your company. It absolutely makes sense to geek out about stuff like this if it's part of your core product. However, if it isn't, then do yourself and your e…

The cost of complication is high. They're optimizing for simplicity. Both for low friction development and easy operation. sqlite is amazingly simple for local dev. Probably even simpler than a json file. High availability Postgres is complex to operate (you need a whole separate strongly consistent data store to coordinate it). Cassandra is also very complex. sqlite is, again, simpler. Litestream makes sqlite plausi…

> sqlite is amazingly simple for local dev

You are buying simpler local dev with a more complicated and error prone production.

> High availability Postgres is complex to operate

High available SQLite is complicated. More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system. Instead, you've got to bolt on solutions to fix the fact that you chose SQLite.

Containers make local dev with postgres or other db technologies pretty slick. I have a hard time buying the "oh no, but then we need to run docker" because docker containers have added benefits often simplifying local dev (All your dev tools defined in the docker file. Done, new dev installs docker and moves from there).

> sqlite is, again, simpler. Litestream makes sqlite plausible for production.

SQLite is a fantastic tool, when it's used in the right location. A backend database is not the right location for SQLite. Litestream may make it "plausible" but it certainly doesn't make it a sane choice.

Re: A database for 2022

#79

Post by mrkurt »

KwisaksHaderach wrote: post #13

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ours…

Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.

Re: A database for 2022

#80

Post by kevin_nisbet »

ramenmeal wrote: post #18

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they…

> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?

This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions.

The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense.

Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.

Post reply on HN