Live data from Hacker News

A database for 2022

tailscale.com

61–70 of 336 posts

Re: A database for 2022

#61

Earlier quoted context omitted.

Okay, but this is a database now. Is there a reason the replication needs to be built-in?

Sorry, I probably didn't phrase it very clearly. I don't think there is anything wrong with needing some external help for that, but the combination of their choices is IMO a bit bizarre: * SQLite which I mostly see as an embeddable database for applications, used on a server. * Single table with key-value pairs. * Having the tool for replication be a quite young project that's in active development.

Plenty of people use sqlite on servers.

They have a single table with KV pairs right now as an artifact of having used the "database with built-in replication which is tailor built for key-value storage" you complain that they're moving away from. The point of the blog post is that they're doing that in part to move away from the single table of KV pairs.

They're not "sharding".

Re: A database for 2022

#62
post #59

So to recap their timeline: * Instead of an actual database use a JSON file. * Write a blog post about how that didn't scale. * Instead of an actual database hand-roll something else. * Write a blog post about how that didn't scale. * Instead of a database with built-in replication which is tailor built for key-value storage use SQLite with a single table containing key-value pairs and some fresh glue to make it repl…

[deleted]

[deleted]

Re: A database for 2022

#63

There's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.

Threads on HN are very sensitive to initial conditions. A very negative (and I think pretty facile) comment took an early lead here, most likely because of its snark/dunk value. Give the thread time, it'll even out.

Re: A database for 2022

#64

They're using litestream [1] to replicate a SQLite database, which streams additions to the SQLite WAL file to object storage and can replay it back. This is fairly hands-off from SQLite's perspective. There's also the SQLite Session Extension [2] that is a built-in way to support generating and applying "patches". I'm curious how these tools will mature, it seems like a good match for microservices. [1]: https://git…

I watched an interview of Mr. Hipp, creator of SQLite, that I can't find now but was pretty interesting. Aside from being way different than I expected for some reason, in a good way - very down to earth and friendly, he was asked specifically about that, and more or less answered that his job was to write a solid DB, and replication can be done elsewhere. That's a pretty honest answer, and looks like someone took up…

He was on an episode of the Changelog podcast in the fall and went into similar topics: https://changelog.com/news/RRAw/visit It's a good listen.

Re: A database for 2022

#65
post #52

Earlier quoted context omitted.

> without containers Why? The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself. Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do)…

Have a look at this headline: > Zero config VPN. Installs on any device in minutes, manages firewall rules for you, and works from anywhere. I don't know how their product works, but my intuition is that your suggestions aren't feasible.

I suggest reading the article. The reason they haven't chosen "boring" tech has nothing to do with the clients running their software and everything to do with their internal development experiences.

> MySQL (or PostgreSQL) would come next. I’m not particularly familiar with anything MySQL post 1998, but I’m sure it would work. The HA story for open source databases is somewhat surprising, though: you can either have traditional lagging replicas, or commit to no-primary-replica clusters that have very surprising transaction semantics. I wasn’t excited about trying to design a stable API or good network graph calculations on top of those semantics. CockroachDB looked very promising, and indeed still does! But it’s relatively new for a database and I was a little concerned about getting attached to features in a fresh DBMS that would be hard to migrate away from if we needed to.

The reason they are ruling out products isn't "this won't work where it runs" its "I don't like this". They talk about worrying about network graphs and proceed to use SQLite and copy garbage around.

Re: A database for 2022

#66
post #46

Earlier quoted context omitted.

Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basicall…

> Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? It depends on what that tool is doing and if the thing it's doing is a core competency for your company. It absolutely makes sense to geek out about stuff like this if it's part of your core product. However, if it isn't, then do yourself and your e…

The cost of complication is high. They're optimizing for simplicity. Both for low friction development and easy operation.

sqlite is amazingly simple for local dev. Probably even simpler than a json file.

High availability Postgres is complex to operate (you need a whole separate strongly consistent data store to coordinate it). Cassandra is also very complex.

sqlite is, again, simpler. Litestream makes sqlite plausible for production.

Re: A database for 2022

#67

I think I'm missing some context. Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision. It seems like something you'd do as a proof of concept or side project. It's interesting that they're blogging about this, as if they're proud of it. I guess I'm just missing the point. This is their 3rd DB Migration, something that I prefer to avoid at all costs. I guess they…

Using a text file and using etcd as a DB for a production system seems like a terrible engineering decision

I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit.

I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years.

It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have.

why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?

Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."

Re: A database for 2022

#68

There's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.

[deleted]

Re: A database for 2022

#70
post #68

There's a lot of negativity here, so I just wanted to comment that I think the approach Tailscale is taking with sqlite is pretty neat. MySQL and Postgres are both pretty complicated, so gluing together two simpler systems, sqlite and litestream, sounds appealing, and I'm interested to hear how it turns out in the long run.

[deleted]

[deleted]
Post reply on HN