Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

71–80 of 190 posts

Re: An Unlikely Database Migration

#71

Post by bradfitz »

jgraettinger1 wrote: post #52
bradfitz wrote: post #35

Earlier quoted context omitted.

> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)

Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change). The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if nee…

Whoa, we hadn't seen that! At first glance it indeed appears to be perhaps exactly identical to what we did.

Re: An Unlikely Database Migration

#72

Post by endymi0n »

judofyr wrote: post #30

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…

This is about spot on. I do get the part about testability, but with a simple Key/Value use case like this, BoltDB or Pebble might have fit extremely well into the Native Golang paradigm as a backing store for the in-memory maps while not needing nearly as much custom code.

Plus maybe replacing the sync.Mutex with RWMutexes for optimum read performance in a seldom-write use case.

On the other hand again, I feel a bit weird criticizing Brad Fitzpatrick ;-) — so there might be other things at play I don‘t have a clue about...

Re: An Unlikely Database Migration

#73

Post by jclulow »

jbverschoor wrote: post #59

Doesn’t sound like smart thing to do and sounds more like a js dev/student discovering step by step why sql databases are so popular.. Probably not so, bc tailscale is a decent product, but this post did not change my view in a good way

On the contrary, it sounds like a seasoned group of people who understand their needs and are wary of the very real challenges presented by most existing SQL systems with respect to deployment, testing, and especially fault tolerance. I'm interested in Cockroach myself, but I also acknowledge it's relatively new, and itself a large and complicated body of software, and choosing it represents a risk.

Re: An Unlikely Database Migration

#75

Post by jgraettinger1 »

irrational wrote: post #34
bradfitz wrote: post #24

Earlier quoted context omitted.

We could've done that. We could've also used DRBD, etc. But then we still have the SQL/ORM/testing latency/dependency problems.

Can you go into more about what these problems are? I've always used databases (about 15 years on Oracle and about 5 years on Postgres) and I'm not sure if I know what problems you are referring to. Maybe I have experienced them, but have thought of them by a different name. SQL - I'm not sure what the problems are with SQL. But it is like a second language to me so maybe I experienced these problems long ago and hav…

I think the "database" label is tripping up the conversation here. What's being talked about here, really, is fast & HA coordination over a (relatively) small amount of shared state by multiple actors within a distributed system. This is literally Etcd's raison d'etre, it excels at this use case.

There are many operational differences between Etcd and a traditional RDBMs, but the biggest ones are that broadcasting updates (so that actors may react) is a core operation, and the MVCC log is "exposed" (via ModRevision) so that actors can resolve state disagreements (am I out of date, or are you?).

Re: An Unlikely Database Migration

#76

Post by nicoburns »

bradfitz wrote: post #65
nicoburns wrote: post #61

Earlier quoted context omitted.

If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).

So then every engineer needs to install & maintain a database on their machines. (Hope they install the right version!) I mean, that's what my old company did pre-Docker. It works, but it's tedious.

I mean that's an `apt install postgres` or `brew install postgres` away. Takes about 5 minutes. I guess it could become a pain if you need to work with multiple different versions at once.

Re: An Unlikely Database Migration

#77

Post by bradfitz »

robmccoll wrote: post #69
bradfitz wrote: post #40

Earlier quoted context omitted.

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) 1 control process every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.

I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble? Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing d…

Yes, we're going to be moving to 2+ control servers for HA + blue/green reasons pretty soon here.

Re: An Unlikely Database Migration

#78

Post by lima »

npiit wrote: post #36
lima wrote: post #29

Earlier quoted context omitted.

It's more zerotrust-y than Cloudflare et al since it's entirely P2P, with only the control plane running in the cloud. Compared to ZeroTier, the Tailscale client has a permissive license, the mesh is fully routed (vs. a L2 network with unencrypted broadcasts), is written in a memory-safe programming language, integrates with company SSO, and uses the Wireguard protocol (i.e. sane, audited crypto instead of a DIY prot…

zerotrust has nothing to do with p2p, zero-trust is about making sure that this user is authorized to access that application at the resource level not using some decades old segmentation/network level policies. Zerotier also claims to be zerotrust but it's technically not. Cloudflare, Citrix, PulseSecure have zerotrust offerings, but many others sadly just claim to be either by ignorance or dishonesty.

Yes, and implementing that is exactly the point of Tailscale, with the added advantage of not relying on a centralized proxy.

Re: An Unlikely Database Migration

#79

Post by majormajor »

nicoburns wrote: post #76
bradfitz wrote: post #65

Earlier quoted context omitted.

So then every engineer needs to install & maintain a database on their machines. (Hope they install the right version!) I mean, that's what my old company did pre-Docker. It works, but it's tedious.

I mean that's an `apt install postgres` or `brew install postgres` away. Takes about 5 minutes. I guess it could become a pain if you need to work with multiple different versions at once.

Being deep in the cloud world right now, with aws and terraform and kubernetes cli tools, etc, not having to install third party tools on my machine does sound pretty great, but also entirely unrealistic.

Managing local DBs once new versions are out and your server isn't upgraded yet is irritating, but when I'm using a Mac I'd still rather use a native DB than Docker because of the VM overhead, since I've not yet run into a bug caused by something like "my local postgres was a different version than the server was." (Closest I've gotten was imagemagick for mac doing something a bit differently than for linux, about 10 years ago at this point.)

Re: An Unlikely Database Migration

#80

Post by aidos »

rubenv wrote: post #15
crawshaw wrote: post #6

Earlier quoted context omitted.

It is a tricky tradeoff for startups. On the one hand, a startup has very limited resources and so has to focus on the business. On the other hand, a startup has to experiment to find the business. I don't think there's an easy answer. In our case, the control plane data store really should be as boring as possible. It was real stretch using anything other than MySQL. We tried to lay out the arguments in the post, bu…

For PostgreSQL and Go, here's a package to spin up a temp in-mem PostgreSQL: https://github.com/rubenv/pgtest/

From memory, in Postgres, you could also have a copy of your base database and then copy it in for each test, which is seem to recall being fairly fast. It includes the data too.

    Create database test_db template initial_db;
Post reply on HN