Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

51–60 of 190 posts

Re: An Unlikely Database Migration

#51

Post by nmjohn »

bradfitz wrote: post #37
irrational wrote: post #34

Earlier quoted context omitted.

Can you go into more about what these problems are? I've always used databases (about 15 years on Oracle and about 5 years on Postgres) and I'm not sure if I know what problems you are referring to. Maybe I have experienced them, but have thought of them by a different name. SQL - I'm not sure what the problems are with SQL. But it is like a second language to me so maybe I experienced these problems long ago and hav…

SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…

Can you put some numbers on how much time is too much? I've never seen anyone go this far to avoid using a database for what sounds like the only "real" reason is to avoid testing latency (a problem which has many other solutions) so I am really confused, but curious to understand!

Re: An Unlikely Database Migration

#52

Post by jgraettinger1 »

bradfitz wrote: post #35
judofyr wrote: post #30

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…

> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)

Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change).

The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if needed) block until your keyspace has caught up such that you read your write -- or someone else's conflicting write.

[0] https://pkg.go.dev/go.gazette.dev/core/keyspace

Re: An Unlikely Database Migration

#55

Post by judofyr »

bradfitz wrote: post #40
judofyr wrote: post #38

Earlier quoted context omitted.

> What do you mean by this part? tailetc is a library used by the client of etcd. Oh. Since they have a full cache of the database I thought it was intended to be used as a separate set of servers layered in front of etcd to lessen the read load. But you're actually using it directly? Interesting. What's the impact on memory usage and scalability? Are you not worried that this will not scale over time since all clien…

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) 1 control process every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.

You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?

Re: An Unlikely Database Migration

#56

Post by crawshaw »

judofyr wrote: post #55
bradfitz wrote: post #40

Earlier quoted context omitted.

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) 1 control process every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.

You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?

We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.

Re: An Unlikely Database Migration

#57

Post by judofyr »

crawshaw wrote: post #56
judofyr wrote: post #55

Earlier quoted context omitted.

You're able to serve all your clients from a single control process? And this would probably work for quite a while? Then I struggle to see why you couldn't just use SQLite. On startup read the full database into memory. Serve reads straight from memory. Writes go to SQLite first and if it succeeds then you update the data in memory. What am I missing here?

We could use SQLite. (I love SQLite and have written about it before!) The goal is N control processes not for scale, but for more flexibility with deployment, canarying, etc.

That makes sense. Thanks for answering all of my critical questions. Looks like a very nice piece of technology you’re building!

Re: An Unlikely Database Migration

#58

Post by apenwarr »

judofyr wrote: post #49
apenwarr wrote: post #28

Earlier quoted context omitted.

That would have been considerably less scalable. etcd has some interesting scaling characteristics. I posted some followup notes on twitter here: https://twitter.com/apenwarr/status/1349453076541927425

How is PostgreSQL (or MySQL) "considerably less scalable" exactly? etcd isn't particularly known for being scalable or performant. I'm sure it's fast enough for your use-case (since you've benchmarked it), but people have been scaling both PostgreSQL and MySQL far beyond what etcd can achieve (usually at the cost of availability of course).

[I work at Tailscale] I only mean scalable for our very specific and weird access patterns, which involves frequently read-iterating through a large section of the keyspace to calculate and distribute network+firewall updates.

Our database has very small amounts of data but a very, very large number of parallel readers. etcd explicitly disclaims any ability to scale to large data sizes, and probably rightly so :)

Re: An Unlikely Database Migration

#59

Post by jbverschoor »

Doesn’t sound like smart thing to do and sounds more like a js dev/student discovering step by step why sql databases are so popular..

Probably not so, bc tailscale is a decent product, but this post did not change my view in a good way

Re: An Unlikely Database Migration

#60

Post by nicoburns »

bradfitz wrote: post #47
mwcampbell wrote: post #44

Earlier quoted context omitted.

> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?

> What if you used one of the managed RDBMS services offered by the big cloud providers? We could (and likely would, despite the costs) but that doesn't address our testing requirements. The control plane is on AWS. We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.

Why is testing Postgres/MySQL difficult? You can easily run a server locally (or on CI) and create new databases for test runs, etc.
Post reply on HN