Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

131–140 of 190 posts

Re: An Unlikely Database Migration

#131

Post by cbushko »

bradfitz wrote: post #122
cbushko wrote: post #117

I think a lot of people are missing the point that a traditional DB (MYSQL/Postgress) are not a good fit for this scenario. This isn't a CRUD application but is instead a distributed control plane with a lot of reads and a small dataset. Joins and complex queries are not needed in this case as the data is simple. I am also going to go out on a limb and guess that this is all running in kubernetes. Running etcd there…

We don't use Kubernetes (or even Docker) currently.

Hopefully tailscale gets to the size where kubernetes is worth it. It's a complex thing to run and understand but in the end I think it is worth it. It has certainly made my day to day life a lot easier and allowed our tiny team to build out a solid platform. It has greatly reduced the amount of time that our developers need to get a service up and running our new features out.

Re: An Unlikely Database Migration

#132

Post by cavisne »

Honestly this feels like engineers that spent too long in FANG and get completely burnt out on dealing with SRE and HA requirements... so decide to built a setup so prone to 2AM pages even a PHP webshop would frown at it.

Curiously though its a pattern I've seen twice in the last 12 months, there was that guide on the good bits of AWS that also recommended starting with a single host with everything running on it.

Maybe we should all move that host back under our desks and really be back to basics!

Re: An Unlikely Database Migration

#133

Post by jbverschoor »

breck wrote: post #41

If you liked this I highly recommend "Clean Architecture" by Uncle Bob. He has a great story of how they kept putting off switching to a SQL DB on a project and then never needed to and it was a big success anyway.

I think uncle bob is the only one who makes sense

Re: An Unlikely Database Migration

#134

Post by nautilus12 »

The premise of articles like this annoys me. It reeks of "we are too smart to use databases", "json is good enough for us", when anyone that works with data to any large extent knows that json is just a pain and we only have to deal with it because the front end is enamored with because "readable" and "javascript".

Re: An Unlikely Database Migration

#135

Post by a1369209993 »

Quekid5 wrote: post #103
miki123211 wrote: post #99

I use the same system (a JSON file protected with a mutex) for an internal tool I wrote, and it works great. For us, file size or request count is not a concern, it's serving a couple (internal) users per minute at peak loads, the JSON is about 150 kb after half a year, and old data could easily be deleted/archived if need be. This tool needs to insert data in the middle of (pretty short) lists, using a pretty compli…

As long as you're guaranteeing correctness[0], it's hard to disagree with the "simple" approach. As long as you don't over-promise or under-deliver, there's no problem, AFAICS. [0] Via mutex in your case. Have you thought about durability, though. That one's actually weirdly difficult to guarantee...

> Have you thought about durability, though. That one's actually weirdly difficult to guarantee...

Strictly speaking, it's literally impossible to guarantee[0], so it's more a question of what kinds and degrees of problems are in- versus out-of-scope for being able to recover from.

0: What happens if I smash your hard drive with a hammer? Oh, you have multiple hard drives? That's fine, I have multiple hammers.

Re: An Unlikely Database Migration

#137

Post by manigandham »

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency."

People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology.

Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or write it yourself (since those objects have to be mapped to the database somehow). 99% of the time, ORMs generate perfectly fine SQL (if not exactly what you'd type anyway) while handling connections, security, mapping, batching, transactions, and other issues inherent in database calls.

The 1% of the time you need something more complex, you can always switch to manual SQL (and ORMs will even let you run that SQL while handling the rest as usual). The overall advantages massively outweigh any negatives, if they even apply to your project.

Re: An Unlikely Database Migration

#138

Post by tgtweak »

cavisne wrote: post #132

Honestly this feels like engineers that spent too long in FANG and get completely burnt out on dealing with SRE and HA requirements... so decide to built a setup so prone to 2AM pages even a PHP webshop would frown at it. Curiously though its a pattern I've seen twice in the last 12 months, there was that guide on the good bits of AWS that also recommended starting with a single host with everything running on it. Ma…

Philosophically: I've seen some "bespoke" systems like this that live long enough for a nice off-the-shelf solution to come around that solves the problem much more elegantly and efficiently than the "bespoke" one does. This seems like a normal and dare I say organic path for these kind of systems to take.

I don't even mind senior devs putting together things like this at the cornerstone of the company provided there are always at any given point in time 2 people that know how it works and can work on it, and sufficient time was spent looking at existing solutions to make that call. It should be made with full expectations that the first paragraph is inevitable.

Specifically, in this case: Without any actual data (# of reads, # of writes, size of writes, size of data, read patterns, consistency requirements) it is not possible to judge whether going custom on such a system was merited or not. I would find it VERY difficult to come to the conclusion that this use case couldn't be solved with very common tooling such as spark and/or nats-streaming. "provided the entire dataset fits in memory" is a very large design liberty when designing such a solution and doesn't scream "scalability" or n+1 node write-consistency to me. I say this acknowledging full well that etcd is an unbelievably well written piece of software with durability and speed beyond it's years.

Keeping my eyes open for that post-series-a-post-mortem post.

Re: An Unlikely Database Migration

#139

Post by strken »

judofyr wrote: post #30

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…

I was initially baffled by the choice of technology too. Part of it is that etcd is apparently much faster at handling writes, and offers more flexibility with regards to consistency, than I remember. Part of it might be that I don't understand the durability guarantees they're after, the gotchas they can avoid (e.g. transactions), or their overall architecture.

Re: An Unlikely Database Migration

#140

Post by MrStonedOne »

manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

Modern ORMs don't let you hand craft sql, shunting it away in some scary extension that you then have to fight when management on using.

A ORM that worked on the principle of insert query text of any complexity receive object as the primary usecase, not the "nonstandard and non-idiomatic usecase" would be the only way to ease the concerns of dba's who code like me.

Its the same pitfall of api clients. Why would I take the time to learn an api like its a sdk, along with the pains of trying to shunt openapi's libraries in to my application without requiring the creation of a composer build step, further complicating deployment, when I can make 5 methods in the time before lunch to do the bits i need as rest queries and deployment of my php app remains as simple as `git pull production` on the nfs share all the workers read from?

The benefit of compile validated symbols is moot in the days of test driven dev, so the benefits gained from that can still be realized without creating build complexity or making competent engineers re-learn something they already know only re-abstracted in a way that almost always makes it harder for somebody who understands the low level to learn the new way compared to a new dev.

Post reply on HN