Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

61–70 of 190 posts

Re: An Unlikely Database Migration

#61

Post by nicoburns »

bradfitz wrote: post #43
irrational wrote: post #39

Earlier quoted context omitted.

Ah, we don't use Docker or any other container technology. Maybe that is why we aren't seeing the latency issues you are referring to.

Docker itself doesn't add much latency. It just makes getting MySQL and PostgreSQL easier. If anything, it helps with dependencies. The database server startup still isn't great, though.

If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).

Re: An Unlikely Database Migration

#62

Post by 256dpi »

I found myself in a similar situation sometime ago with MongoDB. In one project my unit tests started slowing me down too much to be productive. In another, I had so little data that running a server alongside it was a waste of resources. I invested a couple of weeks in developing a SQLite type of library[1] for Go that implemented the official Go drivers API with a small wrapper to select between the two. Up until now, it paid huge dividends in both projects ongoing simplicity and was totally worth the investment.

[1]: https://github.com/256dpi/lungo

Re: An Unlikely Database Migration

#63

Post by bradfitz »

nicoburns wrote: post #60
bradfitz wrote: post #47

Earlier quoted context omitted.

> What if you used one of the managed RDBMS services offered by the big cloud providers? We could (and likely would, despite the costs) but that doesn't address our testing requirements. The control plane is on AWS. We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.

Why is testing Postgres/MySQL difficult? You can easily run a server locally (or on CI) and create new databases for test runs, etc.

It's not difficult. We've done it before and have code for it. See the article.

Re: An Unlikely Database Migration

#64

Post by majormajor »

bradfitz wrote: post #37
irrational wrote: post #34

Earlier quoted context omitted.

Can you go into more about what these problems are? I've always used databases (about 15 years on Oracle and about 5 years on Postgres) and I'm not sure if I know what problems you are referring to. Maybe I have experienced them, but have thought of them by a different name. SQL - I'm not sure what the problems are with SQL. But it is like a second language to me so maybe I experienced these problems long ago and hav…

SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…

Were you doing a lot of logic in SQL itself? Sounds like not really, but then I'm surprised you'd have so many tests hitting the DB directly, vs most feature logic living above that layer in a way that doesn't need the DB running at all.

Re: An Unlikely Database Migration

#65

Post by bradfitz »

nicoburns wrote: post #61
bradfitz wrote: post #43

Earlier quoted context omitted.

Docker itself doesn't add much latency. It just makes getting MySQL and PostgreSQL easier. If anything, it helps with dependencies. The database server startup still isn't great, though.

If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).

So then every engineer needs to install & maintain a database on their machines. (Hope they install the right version!)

I mean, that's what my old company did pre-Docker. It works, but it's tedious.

Re: An Unlikely Database Migration

#67

Post by viraptor »

nmjohn wrote: post #51
bradfitz wrote: post #37

Earlier quoted context omitted.

SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…

Can you put some numbers on how much time is too much? I've never seen anyone go this far to avoid using a database for what sounds like the only "real" reason is to avoid testing latency (a problem which has many other solutions) so I am really confused, but curious to understand!

Not sure what their requirements are, but I'm using a "spin up an isolated postgres instance per test run" solution and end up with ~3s overhead to do that. (Using https://pypi.org/project/testing.postgresql/

Edit: 3s for global setup/teardown. Not per test function/suite.

Re: An Unlikely Database Migration

#68

Post by detaro »

npiit wrote: post #2

Unfortunately this company is known for its shady agressive marketing on hacker news. The upvotes count is really suspicious and this is not the first time. EDIT: Why is the downvoting? the post was given like 9 upvotes in the first 5 minutes. I frequently go to "new" and this is a highly suspicious behaviur.

> EDIT: Why is the downvoting?

Because this kind of thing is something you should contact the mods about, not leave comments that nobody can really (dis-)prove

Re: An Unlikely Database Migration

#69

Post by robmccoll »

bradfitz wrote: post #40
judofyr wrote: post #38

Earlier quoted context omitted.

> What do you mean by this part? tailetc is a library used by the client of etcd. Oh. Since they have a full cache of the database I thought it was intended to be used as a separate set of servers layered in front of etcd to lessen the read load. But you're actually using it directly? Interesting. What's the impact on memory usage and scalability? Are you not worried that this will not scale over time since all clien…

Well, we have exactly 1 client (our 1 control server process). So architecturally it's: 3 or 5 etcd (forget what we last deployed) 1 control process every Tailscale client in the world The "Future" section is about bumping "1 control process" to "N control processes" where N will be like 2 or max 5 perhaps. The memory overhead isn't bad, as the "database" isn't big. Modern computers have tons of RAM.

I'm curious what drove the decision to move to an external store (and multinode HA config at that) now compared to using a local Go KV store like Badger or Pebble?

Given that the goals seem to be improving performance over serializing a set of maps to disk as JSON on every change and keeping complexity down for fast and simple testing, a KV library would seem to accomplish both with less effort, without introducing dependence on an external service, and would enable the DB to grow out of memory if needed. Do you envision going to 2+ control processes that soon?

Any consideration given to running the KV store inside the control processes themselves (either by embedding something like an etcd or by integrating a raft library and a KV store to reinvent that wheel) since you are replicating the entire DB into the client anyway?

Meanwhile I'm working with application-sharded PG clusters with in-client caches with coherence maintained through Redis pubsub, so who am I to question the complexity of this setup haha.

Re: An Unlikely Database Migration

#70

Post by jolux »

bradfitz wrote: post #37
irrational wrote: post #34

Earlier quoted context omitted.

Can you go into more about what these problems are? I've always used databases (about 15 years on Oracle and about 5 years on Postgres) and I'm not sure if I know what problems you are referring to. Maybe I have experienced them, but have thought of them by a different name. SQL - I'm not sure what the problems are with SQL. But it is like a second language to me so maybe I experienced these problems long ago and hav…

SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…

Why are your unit tests touching a database? I’m a real stickler about keeping unit tests isolated, because once I/O gets involved, they invariably become much less reliable and as you mention, too slow.
Post reply on HN