Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A database for 2022

tailscale.com

Post reply on HN
Print view View original

171–180 of 336 posts

Re: A database for 2022

#171

Post by nrr »

jrockway wrote: post #162
tptacek wrote: post #139

Earlier quoted context omitted.

Another comment on this thread tried to dunk on them, saying that they'd done this so many times before (JSON to etcd, etcd to sqlite) that they must be getting really good at it. That comment struck me as so close to understanding what they're doing right here, it was painful.

Yeah. Maybe people are overestimating how expensive this kind of change is. We can all imagine being part of a larger org, where one day an All Hands appears on your calendar, and some exec you've never heard of proclaims the beginning of a 2 year multi-team database migration project. So now instead of making nifty software, you are just in 40 hours of meetings a week fighting over small details on this migration no…

I have the odd feeling that this kind of overestimation comes from the experience that folks tend to have in writing more traditional N-tier applications, especially post-Rails. A certain kind of database-first, ORM-first design philosophy really took hold after that, and it became somewhat common to see incredulity in response to telling someone that you weren't using something like MySQL or Postgres in production.

The kinds of projects that I more generally tend to see make these migrations successfully are the ones that design with in-memory data structures first and then sort out how best to persist that kind of thing to disk after their domain model has solidified a little bit.

Tailscale smells a lot like that to me.

Re: A database for 2022

#172

Post by tptacek »

ransom1538 wrote: post #170
tptacek wrote: post #87

This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…

One of the largest forums online in 2003 was was ezboard. A billion page views per month. Back then - insane - well that was beyond insane. I started work there as a young idiot. "What db you guys using, mysql?...." long pause... "We use a file system." I dug in. Yep. A fucking file system. Files allowed: simple backups, impossible speed, obvious cache systems, partitioning. Later on i met the roomate of the author o…

The guy who wrote memcache is on this thread, I think?

Re: A database for 2022

#173

Post by tptacek »

jeffffff wrote: post #160
ignoramous wrote: post #133

Earlier quoted context omitted.

> is their time really best spent on dba work... It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations). > ...rather than improving the product? Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and h…

> When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason. if zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, why are they using s3? migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted…

First, S3 and a SQL database aren't comparable. But I think you're bringing up S3 because they're using Litestream to ship WAL frames to S3. Go read the Litestream documentation; Litestream syncs to basically anything. They don't need to "migrate to another S3 compatible backend"; they can migrate to almost anything that can save a file.

It's a super confusing argument regardless, because the industry is lousy with "S3-compatible backends".

Re: A database for 2022

#174

Post by ransom1538 »

tptacek wrote: post #172
ransom1538 wrote: post #170

Earlier quoted context omitted.

One of the largest forums online in 2003 was was ezboard. A billion page views per month. Back then - insane - well that was beyond insane. I started work there as a young idiot. "What db you guys using, mysql?...." long pause... "We use a file system." I dug in. Yep. A fucking file system. Files allowed: simple backups, impossible speed, obvious cache systems, partitioning. Later on i met the roomate of the author o…

The guy who wrote memcache is on this thread, I think?

bradfitz lol he is.

Re: A database for 2022

#175

Post by acidbaseextract »

broken8ball wrote: post #165
mathgladiator wrote: post #122

I believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model.. The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated. A document within Adama ( https://www.adama-platform.com/ ) is…

Really curious…how do you meaningfully overlay any indexes on top of a big-ass JSON file? Technical details are appreciated, and no problem if it’s your secret sauce—just very curious how this is accomplished!

The normal way? You can implement whatever kind of index you like — b-tree index, bitmap index, hash index are all useful and conceptually simple if you're familiar with the backing data structures.

For example, if you want to index a "foreign key" id stored in each "record" in a JSON array of objects, you build a hash table from the FK id values to the JSON array indices of the objects that have that id. It can be as stupid simple as an `fk_index = defaultdict(set)` somewhere in your program, to use a Pythonism.

Now when someone wants JSON objects in that array matching a given FK id, they can just O(1) look in the index to know the position of records that match. Much better than an O(N) scan of every item in the array.

Of course you have to to maintain the index as writes to the JSON happen, but that's not bad once you understand how things work. No real secret sauce.

Re: A database for 2022

#176

Post by bobnamob »

odonnellryan wrote: post #109
endisneigh wrote: post #7

I'm so desperate for a SQL database where I can just put it on a bunch of commodity hardware via Docker, connect them up and never worry about this again. Ideally it'd monitor my queries and create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee /dream FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular…

You probably don't really want that. But not in the way that people didn't want WoW classic.

Can you elaborate on why you say this?

Re: A database for 2022

#177

Post by bobnamob »

jeffffff wrote: post #156
tptacek wrote: post #144

Earlier quoted context omitted.

"Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well . We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to…

i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their besp…

The s3 api is rapidly becoming a distributed storage standard. Building your product around it is hardly lock-in these days.

AWS does offer one (very reliable) implementation but they’re very definitely not a monopoly.

Re: A database for 2022

#178

Post by mathgladiator »

broken8ball wrote: post #165
mathgladiator wrote: post #122

I believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model.. The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated. A document within Adama ( https://www.adama-platform.com/ ) is…

Really curious…how do you meaningfully overlay any indexes on top of a big-ass JSON file? Technical details are appreciated, and no problem if it’s your secret sauce—just very curious how this is accomplished!

So, a funny thought experiment is what happens when you parse a JSON file at the same time? You also index by primary key (the field name).

So, I mirror this thinking and having be just an object with the keys being the primary key. Then, I simply index all the children by their fields based on insights from the developer via the index keyword.

So, if you have

record R { public int id; client int owner; int age; index age; } table rows;

then queries for age can be accelerated by the table.

like "iterate rows where age==42" will basically hone in on the bucket of age==42. I currently only index clients by hash and integers.

The critical aspect which makes this work is that I monitor all mutations. When a child object has a field mutated, then it is removed from all indices and placed into an unknown index. Any queries will also consider it as the purpose of queries to simply narrow the field. Once data changes are persisted, the index is updated and items are moved out of the unknown bucket. This works fairly well because the indices are primarily used during the privacy check phase.

Re: A database for 2022

#179

Post by mathgladiator »

acidbaseextract wrote: post #175
broken8ball wrote: post #165

Earlier quoted context omitted.

Really curious…how do you meaningfully overlay any indexes on top of a big-ass JSON file? Technical details are appreciated, and no problem if it’s your secret sauce—just very curious how this is accomplished!

The normal way? You can implement whatever kind of index you like — b-tree index, bitmap index, hash index are all useful and conceptually simple if you're familiar with the backing data structures. For example, if you want to index a "foreign key" id stored in each "record" in a JSON array of objects, you build a hash table from the FK id values to the JSON array indices of the objects that have that id. It can be a…

The secret sauce may be the need to take control of the write path.

Re: A database for 2022

#180

Post by avinassh »

jlokier wrote: post #97
ignoramous wrote: post #82

Earlier quoted context omitted.

Tailscale mention they want to run tests locally, not deal with MySQL/Postgres ops overhead themselves, and also avoid vendor lock-in (rules out RDS / Aurora, PlanetScale, CockroachDB, YugaByte, Spanner et al). May be Oracle remains an option. ;)

I don't see how CockroachDB ends up in your list of DBs that lead to vendor lock-in or being unable to run a cluster locally, because it's open source and you can run your own cluster. Is there something about it I haven't spotted that puts it in that list?

CockroachDB is licensed under BSL, so it is not really open source rather a source open or source available project. I checked their LICENSE file [0] and it mentions bunch of other open source licenses with BSL. I randomly checked a code file [1] and it seems the code is licensed under BSL.

[0] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/LICENS...

[1] - https://github.com/cockroachdb/cockroach/blob/2c4e2c6/pkg/kv...

Post reply on HN