Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A database for 2022

tailscale.com

Post reply on HN
Print view View original

111–120 of 336 posts

Re: A database for 2022

#111

Post by billywhizz »

silisili wrote: post #14
infogulch wrote: post #2

They're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fairly hands-off from SQLite's perspective. There's also the SQLite Session Extension [2] that is a built-in way to support generating and applying "patches". I'm curious how these tools will mature, it seems like a good match for microservices. [1]: https://git…

I watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up…

probably this one? https://corecursive.com/066-sqlite-with-richard-hipp/

Re: A database for 2022

#112

Post by dahfizz »

This should be a new logical fallacy: appeal to current year. Every time I hear someone say "it's $CURRENT_YEAR, we obviously need to implement $PARTISAN_POLICY" I wonder about what fringe politics will be "obviously correct" in 3022.

Re: A database for 2022

#113

Post by [deleted] »

dahfizz wrote: post #112

This should be a new logical fallacy: appeal to current year. Every time I hear someone say "it's $CURRENT_YEAR, we obviously need to implement $PARTISAN_POLICY" I wonder about what fringe politics will be "obviously correct" in 3022.

[deleted]

Re: A database for 2022

#114

Post by ignoramous »

tptacek wrote: post #87

This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…

> This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it.

To the forum's credit, the top comment (by u/judofyr) the last time tailscale blogged about their unusal db setup, got it exactly right: https://news.ycombinator.com/item?id=25768042

> Interesting choice of technology, but [tailscale] didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica.

> In particular [tailscale has] designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd. That's quite a brave task considering how many databases fail at Jepsen.

> Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten [tailscale] quite far.

Re: A database for 2022

#115

Post by AnimalMuppet »

dahfizz wrote: post #112

This should be a new logical fallacy: appeal to current year. Every time I hear someone say "it's $CURRENT_YEAR, we obviously need to implement $PARTISAN_POLICY" I wonder about what fringe politics will be "obviously correct" in 3022.

Seconded. It's a logical fallacy because they are using the year as proof that we should adopt their policy. Except the year only proves what year it is - nothing else.

Re: A database for 2022

#116

Post by lmeyerov »

kevin_nisbet wrote: post #80
ramenmeal wrote: post #18

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they…

> But I'm just confused, why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions? This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of…

There's a world of a difference of someone doing that for a Google scale thing with big teams supporting it, or say a core part of making the best wireguard product that only they can do, vs a system that could have been postgres and now when the authors leave for their own sqlite-by-mail startup or are partying at SREConf for the 'our n+1th attempt to not do postgres' and an outage causes an unnecessary and ungoogleable shitshow for their colleagues.

A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)

Re: A database for 2022

#117

Post by tptacek »

cogman10 wrote: post #81
tptacek wrote: post #57

Earlier quoted context omitted.

They picked (checks notes) sqlite. It's quite possibly the most boring choice available!

And they tacked on Litestream, which made for far more interesting failure modes. Hopefully they ran their jepson... oh wait, no they didn't really care about the reasons they ruled out traditional db choices.

Do you understand how Litestream works? What's the Jepson test you'd do here, to make sure your single, statically assigned leader isn't conflicting with... itself? All Litestream does is ship WAL segments. The whole point of this design is not having the distributed consensus problems (and concomitant schema and development inflexibility) that etcd has.

Re: A database for 2022

#119

Post by jeffffff »

mrkurt wrote: post #79
KwisaksHaderach wrote: post #13

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ours…

Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.

they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.

Re: A database for 2022

#120

Post by tptacek »

cogman10 wrote: post #78
mrkurt wrote: post #66

Earlier quoted context omitted.

The cost of complication is high. They're optimizing for simplicity. Both for low friction development and easy operation. sqlite is amazingly simple for local dev. Probably even simpler than a json file. High availability Postgres is complex to operate (you need a whole separate strongly consistent data store to coordinate it). Cassandra is also very complex. sqlite is, again, simpler. Litestream makes sqlite plausi…

> sqlite is amazingly simple for local dev You are buying simpler local dev with a more complicated and error prone production. > High availability Postgres is complex to operate High available SQLite is complicated. More complicated by the fact that SQLite wasn't designed at all around the notion of being a multi-tenant database system. Instead, you've got to bolt on solutions to fix the fact that you chose SQLite.…

They don't have a multi-tenant database problem! This is the second weird thing you've said about this design (the other was that they'd need to run Jepson simulations for it). They're not a database hosting provider; they're the single producer and the single consumer of their database. They have a single writing leader, and read-only replicas. That's how Litestream works; it's the only way it works.
Post reply on HN