Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A database for 2022

tailscale.com

Post reply on HN
Print view View original

91–100 of 336 posts

Re: A database for 2022

#91

Post by infogulch »

benbjohnson wrote: post #77
chubot wrote: post #38

Honest question: what do the sqlite authors think of litestream? I seem to recall Richard Hipp on the "changelog" podcast mentioning it, but I don't remember what he said. 2 episodes here: https://changelog.com/person/drh/podcasts#feed I think it was very neutral, something to the effect of "there are multiple solutions". But I know essentially nothing about sqlite internals so I can't judge, or maybe that's why I di…

> Honest question: what do the sqlite authors think of litestream? Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff tha…

Wow that's great that you were able to discuss with the SQLite team directly. Did you discuss how litestream compares to the session extension?

Re: A database for 2022

#92

Post by bccdee »

cogman10 wrote: post #73
dan-robertson wrote: post #56

Earlier quoted context omitted.

Doesn’t the blog post explain why they chose this step rather than doing the thing you suggest? In particular I don’t see how this comment really addresses any of their arguments. And if they got on ok with the json file, maybe you should update towards the features of a complicated RDBMS not being worth the cost?

No, it actually doesn't. The blog post explains "Hey, we've chosen a bunch of crazy ways to store data on the backend. We looked at non-crazy ways, but decided instead it'd be more fun to rsync SQLite data around because SQLite is my favorite DB!" > MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is…

Sqlite replication via litestream has bascially no overhead. It's a great choice if your database only needs to service one client.

It sounds like you're essentially arguing that people should always use the most robust data store available regardless of cost or requirements. Their needs were served just fine by a json file ­for quite some time — if that's all you need, then maintaining a full Posgres cluster is the "crazy" solution actually.

Re: A database for 2022

#93

Post by bestinterest »

nosefrog wrote: post #53

There's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.

Agreed, it's a bit of a shame for HN imo. I see a team who is being brilliant and understanding their domain down to the core with as minimal resources as they can get away with.

There is incredible value in incrementally adding complexity only when needed.

Re: A database for 2022

#94

Post by benbjohnson »

infogulch wrote: post #91
benbjohnson wrote: post #77

Earlier quoted context omitted.

> Honest question: what do the sqlite authors think of litestream? Litestream author here. Dr Hipp and his team reached out when I first released Litestream and we had a video call together. They were fantastic. Really friendly and down to earth. I explained how I put together Litestream and we went back and forth on different approaches. They were really helpful with understanding some of the shared memory stuff tha…

Wow that's great that you were able to discuss with the SQLite team directly. Did you discuss how litestream compares to the session extension?

We didn’t discuss the session extension. That extension solves a different problem of having multiple primaries that need to merge their data. I haven’t used it personally but it looks great for that purpose.

Re: A database for 2022

#95

Post by ignoramous »

epolanski wrote: post #84
brycelarkin wrote: post #43

It looks like you guys use AWS. Why not just use RDS if you don’t want to deal with database management?

Some engineering teams seem to take overcomplication always one step too far. It's very hard to estimate future work and overconfident engineers consistently downplay the costs.

I'm not sure why you're being downvoted. You're right: running databases isn't for every team! Reach out for a managed database if you can.

But tailscale isn't some random group of engs. They've probably got the chops to pull off literally anything they want to. I mean TFA casually mentions online cross-database transfers, multiple zero-downtime schema migrations, inspecting litestream's replication code for feasibility, deftly modifying sqlite WAL checkpoints... all in one breath.

Re: A database for 2022

#96

Post by jitl »

norom wrote: post #23

I for one am enjoying this. Sure they could probably just use a regular ol' database, but where is the excitement in that!

SQLite is the most deployed database in the world

Re: A database for 2022

#97

Post by jlokier »

ignoramous wrote: post #82
AtlasBarfed wrote: post #10

... this is just replication of a database? As in the full database per node/replica? Hasn't AWS's SQL db-as-a-services had this for years now?

Tailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al). May be Oracle remains an option. ;)

I don't see how CockroachDB ends up in your list of DBs that lead to vendor lock-in or being unable to run a cluster locally, because it's open source and you can run your own cluster. Is there something about it I haven't spotted that puts it in that list?

Re: A database for 2022

#98

Post by mardifoufs »

bradfitz wrote: post #34
ramenmeal wrote: post #18

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they…

Tailscale engineer here. > Using a text file and ... > It seems like something you'd do as a proof of concept I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.

No post body was provided.

Re: A database for 2022

#99

Post by ignoramous »

jlokier wrote: post #97
ignoramous wrote: post #82

Earlier quoted context omitted.

Tailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al). May be Oracle remains an option. ;)

I don't see how CockroachDB ends up in your list of DBs that lead to vendor lock-in or being unable to run a cluster locally, because it's open source and you can run your own cluster. Is there something about it I haven't spotted that puts it in that list?

Ops overhead would be my guess.

Take a look at https://litestream.io/guides/systemd/ and compare that with CockroachDB's.

In all fairness, litestream is no CockroachDB/PlanetScale/YugaByte; but why pay the ops overhead (and your lawyers to make sense of their "OSS" licenses) when litestream would do just as nicely.

TFA goes (emphasis mine):

Litestream..grabs a lock so that no other process can checkpoint. It then watches the WAL file and streams the appended blocks up to S3, periodically checkpointing the database for you when it has the necessary segments uploaded. ___This gives you near real time backups and lets you replay your database from S3 trivially..It's a great hack, and a small enough program that I could read my way through it entirely before committing to it.___

Re: A database for 2022

#100

Post by rablackburn »

I loved this article, despite the negativity.

A discussion that keeps recurring on HN is how you can get a lot done with simple, boring solutions. This is a perfect example of that in practice. What’s the simplest thing that’ll work? Do that, monitor the solution, and when you start hitting a limitation reevaluate what the next appropriate solution is.

There are plenty of comments here deriding this as insanity when we have established best practices, but “best practices for me are not necessarily best practices thee”.

People aren’t perfect, not every engineer comes fully-formed grokking the industry’s best solutions and their trade offs.

This is what learning looks like. This is what an evolving system looks like. And this is how valuable software gets built every day.

Who gives a fuck about scalability on a poc? “Let’s just put in a json file”. I Love it. Personally I start at the “use SQLite” step and go from there because SQLite is such amazing software. Thanks to this post I’m excited to try out Lifestream.

You might rightly point out that Tailscale isn’t a side-project/poc anymore. But that’s exactly why they’re changing to SQLite and writing this post; this is their team learning to deal with larger problems and changing needs, and they’re sharing that journey with us.

As long as they’re not _ignoring_ serious risks let them experiment. But try not to get too upset if they’re making different risk/effort trade-offs than you would. Even if it’s “objectively” the “wrong” thing.

My rubric for technical sanity is:

- are you taking (and testing) backups?

- is your infra/data secure?

- are you monitoring what’s happening?

- can you recover from catastrophic failure in a acceptable time frame?

- are you meeting your legal obligations?

Cover those needs and you can be forgiven a lot suboptimal/experimental implementation details.

Post reply on HN