... this is just replication of a database? As in the full database per node/replica? Hasn't AWS's SQL db-as-a-services had this for years now?
Tailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al). May be Oracle remains an option. ;)
A database for 2022
121–130 of 336 posts
Re: A database for 2022
#122The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated.
A document within Adama (https://www.adama-platform.com/) is basically a giant JSON file held within memory with clients connected via a WebSocket. I'm basically building my own indexing since I want the indexing to be reactive, and I've got exceptionally fun optimization problems.
Litestream, in my opinion, is a great way to get started. Actually, it's beyond fantastic if you maintain the 1 server to 1 database because migrations become so easy. So, I applaud the team for taking this simple approach.
Re: A database for 2022
#123Earlier quoted context omitted.
Tailscale employee here. Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property. We have MySQL and PostgreSQL veterans on the team. We know those options well.
> without containers Why? The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself. Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do)…
At some point, migrations were taking too long, so I just started caching the database schema. I have a go generate thingie that creates a fresh database, applies all the migrations, and dumps the schema to a file that gets checked in with the code. (CI checks that the migrations actually result in the database state that's checked in. Yes, there is something that regexes out the hostname from the sql file, since Postgres dumps that in there by default and that results in spurious diffs.)
I have learned that people often put up with really sub-optimal developer workflows. My test is that you should be able to add "print hello world" to the top of your main function, and see that printed on your screen in less than 5 seconds. (Yes, even if your application is complicated. I have a hacked up copy of K8s that starts that quickly, so that you can write an app that deeply integrates with K8s and not have to reuse a cluster between tests, or pay a long cluster creation cost. All obstacles must be removed!) If it takes longer than that, you are just throwing your developer's salaries into the toilet. (If it takes 6 seconds, you open up HN, and there's an hour gone!)
Re: A database for 2022
#124Earlier quoted context omitted.
Some engineering teams seem to take overcomplication always one step too far. It's very hard to estimate future work and overconfident engineers consistently downplay the costs.
I'm not sure why you're being downvoted. You're right: running databases isn't for every team! Reach out for a managed database if you can. But tailscale isn't some random group of engs. They've probably got the chops to pull off literally anything they want to. I mean TFA casually mentions online cross-database transfers, multiple zero-downtime schema migrations, inspecting litestream's replication code for feasibil…
Re: A database for 2022
#125Earlier quoted context omitted.
Do you want to know how popular systems are actually built, from the inside, or do you want carefully groomed triumphal announcements of new, perfectly-formed features? I struggle with this a lot in my writing too, but I've drawn a conclusion I'm sticking with: it's better to relay what the team is actually doing, "warts" and all. We ranked on HN a few weeks ago with a post about user-mode WireGuard that was basicall…
I love Tailscale and it's true that I am not suffering from stability problems with its database choices. But, I do have challenges working with their ACL structure -- which is backed by their database -- and specifically would love to be able to update users/groups separately from tag groups and both of those separately from ACL rules and I can't help but wonder if the fact that this feature doesn't exist relates to…
Unrelated.
We just haven't worked on it much. A bunch of other stuff has been a higher priority. But better ACL management will happen.
Re: A database for 2022
#126Earlier quoted context omitted.
If you're dinging them for not using Cockroach, they explained why they're not using Cockroach in the exact paragraph you quoted: if they use Cockroach, they're committing to the scaling and distribution system that Cockroach provides, which might or might not be a good fit for them 2 years from now. If it isn't a good fit, they're stuck with a huge engineering bill to get themselves out of Cockroach. We're moving to…
> If it isn't a good fit, they're stuck with a huge engineering bill to get themselves out of Cockroach. Similar to the current engineering bills they are paying because they switched from a json file -> etcd -> sqlite? Seems like the cost of switching isn't really an issue for them since they've pulled that lever multiple times now.
Re: A database for 2022
#127This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…
This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then everyone would be like "why didn't you just scp a json file between machines" ;)
Re: A database for 2022
#128I believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model.. The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated. A document within Adama ( https://www.adama-platform.com/ ) is…
(we don't use Litestream right now; we're pushing the complexity up a layer in our design instead)
A lot of people on this thread are looking down their noses at sqlite, but I think they're kind of beclowning themselves; the unreasonable effectiveness of sqlite has been a meme in infra dev for a couple years now. It's not a new idea. Lots of people are doing stuff like this.
The funniest bit on this thread is the person saying they should use RDS, as if their infrastructure was just a big Rails app.
Re: A database for 2022
#129I'm so desperate for a SQL database where I can just put it on a bunch of commodity hardware via Docker, connect them up and never worry about this again. Ideally it'd monitor my queries and create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee /dream FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular…
Re: A database for 2022
#130This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…
My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack.
I know there are limits to MySQL but so far in my career I haven't had any projects hit those limits because hardware has been getting better faster.