Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

121–130 of 190 posts

Re: An Unlikely Database Migration

#121

Post by xupybd »

bradfitz wrote: post #84
nmjohn wrote: post #51

Earlier quoted context omitted.

Can you put some numbers on how much time is too much? I've never seen anyone go this far to avoid using a database for what sounds like the only "real" reason is to avoid testing latency (a problem which has many other solutions) so I am really confused, but curious to understand!

Running all of our control server tests (including integration tests) right now takes 8 seconds, and we're not even incredibly happy with that. There's no reason it should even be half that. So it's not really in our patience budget for adding a mysqld or postgres start up (possible docker pull, create its schema, etc).

Wow that's really quick. I can see why that would be very desirable.

It would create a nice flow to get feedback from your test suite that quickly.

Re: An Unlikely Database Migration

#122

Post by bradfitz »

cbushko wrote: post #117

I think a lot of people are missing the point that a traditional DB (MYSQL/Postgress) are not a good fit for this scenario. This isn't a CRUD application but is instead a distributed control plane with a lot of reads and a small dataset. Joins and complex queries are not needed in this case as the data is simple. I am also going to go out on a limb and guess that this is all running in kubernetes. Running etcd there…

We don't use Kubernetes (or even Docker) currently.

Re: An Unlikely Database Migration

#123

Post by segmondy »

judofyr wrote: post #30

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…

if you want a distributed key/value data store, you want to use what's already out there and vetted. It use to be zookeeper, but etcd is much simpler and that's what Kubernetes uses and it has been a big success and proved itself out there in the field. Definitely easier than a full SQL database which is overkill and much harder to replicate especially if you want to have a cluster of >= 3. Again, key is "distributed" and that immediately rules out sqlite.

Re: An Unlikely Database Migration

#124

Post by rhizome »

jamescun wrote: post #3

This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup whose goal is to innovate a techology. Feels demotivating as an engineer if management says our team can only innovate an arbitrary N times.

Every decision to increase platform spread should be justified in detail, as the implementation and support overhead is essentially unbounded. Be careful you don't buy a pig in a poke.

Re: An Unlikely Database Migration

#125

Post by lmilcin »

petters wrote: post #87

> (Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency.) Loss of efficiency? Come on, you were using a file before! :-) Article makes me glad I'm using Django. Just set up a managed Postgres instance in AWS and be done with it. Sqlite for testing locally. Just works and very little engineering time spent on persistent storage. Note: I d…

Efficiency can be measured in many different ways.

Having no dedicated database server or even database instance, being able to persist data to disk with almost no additional memory required, marginal amount of CPU and no heavy application dependencies can be considered very efficient depending on context.

Of course, if you start doing this on every change, many times a second, then it stops being efficient. But then there are ways to fix it without invoking Oracle or MongoDB or other beast.

When I worked on algorithmic trading framework the persistence was just two pointers in memory pointing to end of persisted and end of published region. Occasionally those pointers would be sent over to a dedicated CPU core that would be actually the only core talking to the operating system, and it would just append that data to a file and publish completion so that the other core can update the pointers.

The application would never read the file (the latency even to SSD is such that it could just as well be on the Moon) and the file was used to be able to retrace trading session and to bring up the application from event log in case it failed mid session.

As the data was nicely placed in order in the file, the entire process of reading that "database" would take no more than 1,5s, after which the application would be ready to synchronize with trading session again.

Re: An Unlikely Database Migration

#127

Post by throwdbaaway »

apenwarr wrote: post #58
judofyr wrote: post #49

Earlier quoted context omitted.

How is PostgreSQL (or MySQL) "considerably less scalable" exactly? etcd isn't particularly known for being scalable or performant. I'm sure it's fast enough for your use-case (since you've benchmarked it), but people have been scaling both PostgreSQL and MySQL far beyond what etcd can achieve (usually at the cost of availability of course).

[I work at Tailscale] I only mean scalable for our very specific and weird access patterns, which involves frequently read-iterating through a large section of the keyspace to calculate and distribute network+firewall updates. Our database has very small amounts of data but a very, very large number of parallel readers. etcd explicitly disclaims any ability to scale to large data sizes, and probably rightly so :)

This is getting confusing. The tweets sound like you are concerned about write scalability, and here it sounds like you are concerned about read scalability?

> So we can get, say, 1000 updates, bundle them all, get it synced in say ~100ms, and then answer all 1000 requests at once, and still only take ~100ms.

I assume the same trick is applicable to RDBMS as well? So you batch the 1000 updates, and do one commit with a single fsync.

> Virtually every other database I've used is quite naive about how they flush blocks to disk, which dramatically reduces their effective transactions/sec. It's rare to see one that made all the right choices here.

Can you elaborate on this? Anyway RDBMS worth its salt should be able to saturate the disk IOPS, i.e. the act of flushing itself wouldn't be the bottleneck.

> Our database has very small amounts of data but a very, very large number of parallel readers.

So the control plane is the sole writer of this database, and there are maybe 100s/1000s of other readers, who each has a watcher on etcd? Who are these readers? If they are different processes on different machines, how did it work when the database was in the json file?

Sorry for the barrage of questions, but I have to ask out of curiosity.

Re: An Unlikely Database Migration

#128

Post by jgraettinger1 »

staunch wrote: post #109
crawshaw wrote: post #82

Earlier quoted context omitted.

Drat! I went looking for people doing something similar when I sat down to design our client, but did not find your package. That's a real pity, I would love to have collaborated on this. I guess Go package discovery remains an unsolved problem.

> I guess Go package discovery remains an unsolved problem. Or did you just not really search, like most of us excited to DIY? :-D Godoc is pretty good, the package shows up for the searches I'd probably do in a similar situation. https://godoc.org/?q=etcd https://godoc.org/?q=etcd+watch

Funny, the package in question exists because _I_ thought I could do better and wanted to DIY.

Re: An Unlikely Database Migration

#129

Post by staunch »

jgraettinger1 wrote: post #128
staunch wrote: post #109

Earlier quoted context omitted.

> I guess Go package discovery remains an unsolved problem. Or did you just not really search, like most of us excited to DIY? :-D Godoc is pretty good, the package shows up for the searches I'd probably do in a similar situation. https://godoc.org/?q=etcd https://godoc.org/?q=etcd+watch

Funny, the package in question exists because _I_ thought I could do better and wanted to DIY.

You must not have seen https://godoc.org/?q=tailetc

Re: An Unlikely Database Migration

#130

Post by harikb »

jgraettinger1 wrote: post #52
bradfitz wrote: post #35

Earlier quoted context omitted.

> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)

Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change). The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if nee…

I wish pkg.dev had a signin and option to star/watch a package. I do this with GitHub repos I should revisit. Would have been handy for pkg.dev :) yes, I know - nobody wants yet another login
Post reply on HN