Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

91–100 of 190 posts

Re: An Unlikely Database Migration

#91

Post by bradfitz »

jolux wrote: post #70
bradfitz wrote: post #37

Earlier quoted context omitted.

SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…

Why are your unit tests touching a database? I’m a real stickler about keeping unit tests isolated, because once I/O gets involved, they invariably become much less reliable and as you mention, too slow.

Sorry, I should've just said tests. Our tests overall are a mix of unit tests and integration tests and all sizes in between, depending on what they want to test.

Re: An Unlikely Database Migration

#92

Post by bradfitz »

worik wrote: post #86

This jumped out at me: "The obvious next step to take was to move to SQL" No. Not unless your data is relational. This is a common problem, relational databases have a lot of over head. They are worth it when dealing with relational data. Not so much with non relational data.

Maybe "obvious" was the wrong word there. What we meant to say was that would be the typical route people would go, doing the common, well-known, risk-averse thing.

Re: An Unlikely Database Migration

#93

Post by ZephyrBlu »

sbierwagen wrote: post #50
jamescun wrote: post #3

This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup whose goal is to innovate a techology. Feels demotivating as an engineer if management says our team can only innovate an arbitrary N times.

What is the purpose of a startup? Is it to put keywords on your resume, or is it to create a product?

Depends if you have funding or not.

Re: An Unlikely Database Migration

#95

Post by bradfitz »

shapiro92 wrote: post #94

I find the article a bit hard to follow. What were the actual requirements? I probably didnt understand all of this, but was the time spent on thinking about this more valuable than using a KV Store?

Yeah, I guess we could've laid that out earlier:

* our data is tiny and fits in RAM

* our data changes often

* we want to eventually get to an HA setup (3-5 etcd nodes now, a handful of backend server instances later)

* we want to be able to do blue/green deploys of our backend control server

* we want tests to run incredibly quickly (our current 8 seconds for all tests is too slow)

* we don't want all engineers to have to install Docker or specific DB versions on their dev machines

Re: An Unlikely Database Migration

#96

Post by bradfitz »

mwcampbell wrote: post #83
bradfitz wrote: post #21

Earlier quoted context omitted.

I wrote about that in the article. Search for "dockertest". We considered that option but didn't like it. It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees.

> It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees. Did you configure the Postgres (or MySQL) database to be entirely in memory, e.g. by using a tmpfs Docker volume? As for being onerous or annoying for new employees, which is worse: having to set up a Docker environment, or using a relatively obscure data store in a way that nobody else does?

Empirically, the former.

We've since hired many employees who just learned about our database today from this blog post but had been happily testing against it on their laptops for months.

3-4 of us know about it, and that's sufficient.

Re: An Unlikely Database Migration

#97

Post by jbverschoor »

jclulow wrote: post #73
jbverschoor wrote: post #59

Doesn’t sound like smart thing to do and sounds more like a js dev/student discovering step by step why sql databases are so popular.. Probably not so, bc tailscale is a decent product, but this post did not change my view in a good way

On the contrary, it sounds like a seasoned group of people who understand their needs and are wary of the very real challenges presented by most existing SQL systems with respect to deployment, testing, and especially fault tolerance. I'm interested in Cockroach myself, but I also acknowledge it's relatively new, and itself a large and complicated body of software, and choosing it represents a risk.

Building your own locking mechanism to dump out a JSON file is exactly how you would do this.

> Through this process we would do major reorganizations of our SQL data model every week, which required an astonishing amount of typing. SQL is widely used, durable, effective, and requires an annoying amount of glue to bring into just about any programming language

and

> So we invested what probably amounts to two or three weeks of engineering time into designing in-memory indexes that are transactionally consistent

Sounds to me like someone has learned a lot on the job. Good for him, but it looks exactly like what I said before.

Re: An Unlikely Database Migration

#98

Post by CleverLikeAnOx »

breck wrote: post #41

If you liked this I highly recommend "Clean Architecture" by Uncle Bob. He has a great story of how they kept putting off switching to a SQL DB on a project and then never needed to and it was a big success anyway.

I thoroughly enjoyed "Clean Architecture." I engage with all of Uncle Bob's work in an adversarial manner: he is trying to convince me that he is right and I am playing devil's advocate. This helped me gain insight from his work, develop my own thoughts on architecture, and avoid getting too caught up in doing things the "one true way."

Re: An Unlikely Database Migration

#99

Post by miki123211 »

I use the same system (a JSON file protected with a mutex) for an internal tool I wrote, and it works great. For us, file size or request count is not a concern, it's serving a couple (internal) users per minute at peak loads, the JSON is about 150 kb after half a year, and old data could easily be deleted/archived if need be.

This tool needs to insert data in the middle of (pretty short) lists, using a pretty complicated algorithm to calculate the position to insert at. If I had used an RDBMS, I'd probably have to implement fractional indexes, or at least change the IDs of all the entries following the newly inserted one, and that would be a lot of code to write. This way, I just copy part of the old slice, insert the new item, copy the other part (which are very easy operations in Go), and then write the whole thing out to JSON.

I kept it simple, stupid, and I'm very happy I went with that decision. Sometimes you don't need a database after all.

Re: An Unlikely Database Migration

#100

Post by ropable »

Tangentially, this article makes me so very glad that our own work projects are all making use of the Django ORM. Database migrations and general usage have been a non-issue for literally years.
Post reply on HN