Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

141–150 of 190 posts

Re: An Unlikely Database Migration

#141

Post by bradfitz »

cbushko wrote: post #131
bradfitz wrote: post #122

Earlier quoted context omitted.

We don't use Kubernetes (or even Docker) currently.

Hopefully tailscale gets to the size where kubernetes is worth it. It's a complex thing to run and understand but in the end I think it is worth it. It has certainly made my day to day life a lot easier and allowed our tiny team to build out a solid platform. It has greatly reduced the amount of time that our developers need to get a service up and running our new features out.

We have a lot of Kubernetes experience on the team. Multiple of us run Kubernetes clusters in our home labs (mine: https://github.com/bradfitz/homelab), and one of us used to be on the Google GKE team as an SRE, and is the author of https://metallb.universe.tf/ (which multiple of us also use).

Us _not_ using Kubernetes isn't because we don't know how to use it. It's because we _do_ know how to use it and when _not_ to use it. :)

Re: An Unlikely Database Migration

#142

Post by bradfitz »

staunch wrote: post #129
jgraettinger1 wrote: post #128

Earlier quoted context omitted.

Funny, the package in question exists because _I_ thought I could do better and wanted to DIY.

You must not have seen https://godoc.org/?q=tailetc

Well, we literally named and pushed that publicly minutes before today's blog post.

Re: An Unlikely Database Migration

#143

Post by manigandham »

MrStonedOne wrote: post #140
manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

Modern ORMs don't let you hand craft sql, shunting it away in some scary extension that you then have to fight when management on using. A ORM that worked on the principle of insert query text of any complexity receive object as the primary usecase, not the "nonstandard and non-idiomatic usecase" would be the only way to ease the concerns of dba's who code like me. Its the same pitfall of api clients. Why would I tak…

You're not using a modern ORM then if you can't use manual SQL queries. And tests are not a replacement for compilation and type-safety.

Like I said, you're either using an existing ORM or just writing your own everytime, and the one you write probably won't be very good as seen by the numerous security and performance bugs that are constantly found. Also abstractions are useful. All of software development is built with abstractions and they don't suddenly become useless when it comes to databases.

I also don't see what this has to do with API clients or build steps, but there are good and bad examples of those too.

Re: An Unlikely Database Migration

#144

Post by MrStonedOne »

manigandham wrote: post #143
MrStonedOne wrote: post #140

Earlier quoted context omitted.

Modern ORMs don't let you hand craft sql, shunting it away in some scary extension that you then have to fight when management on using. A ORM that worked on the principle of insert query text of any complexity receive object as the primary usecase, not the "nonstandard and non-idiomatic usecase" would be the only way to ease the concerns of dba's who code like me. Its the same pitfall of api clients. Why would I tak…

You're not using a modern ORM then if you can't use manual SQL queries. And tests are not a replacement for compilation and type-safety. Like I said, you're either using an existing ORM or just writing your own everytime, and the one you write probably won't be very good as seen by the numerous security and performance bugs that are constantly found. Also abstractions are useful. All of software development is built…

> I also don't see what this has to do with API clients

They are the same thing, a way of taking text based queries for formatted data as std:maps and shunting that away behind classes and types and interfaces.

> Like I said, you're either using an existing ORM or just writing your own everytime

vectors of maps works pretty well.

Re: An Unlikely Database Migration

#145

Post by mindwok »

segmondy wrote: post #123
judofyr wrote: post #30

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…

if you want a distributed key/value data store, you want to use what's already out there and vetted. It use to be zookeeper, but etcd is much simpler and that's what Kubernetes uses and it has been a big success and proved itself out there in the field. Definitely easier than a full SQL database which is overkill and much harder to replicate especially if you want to have a cluster of >= 3. Again, key is "distributed…

It's overkill until it's not. We chose etcd initially but after a while we started wanted to ask questions about our data that weren't necessarily organised in the same way as the key/value hierarchy. That just moved all the processing to the client, and now I just wish we used a SQL database from the beginning.

Re: An Unlikely Database Migration

#146

Post by nine_k »

manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

IMHO, ORMs are a wrong abstraction. Full-fledged objects are a wrong abstraction of the data in more cases than not.

The right tool is a wrapper / DSL over SQL, which allows to interact with the database in a predictable and efficient way, while not writing placeholder-ridden SQL by hand. Composable, typesafe, readable.

ORMs do fine in small applications without performance requirements. The further you go from that, the less adequate an ORM becomes, and the more you have to sidestep and even fight it, in my experience.

Re: An Unlikely Database Migration

#147

Post by manigandham »

nine_k wrote: post #146
manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

IMHO, ORMs are a wrong abstraction. Full-fledged objects are a wrong abstraction of the data in more cases than not. The right tool is a wrapper / DSL over SQL, which allows to interact with the database in a predictable and efficient way, while not writing placeholder-ridden SQL by hand. Composable, typesafe, readable. ORMs do fine in small applications without performance requirements. The further you go from that,…

Do you have an example of the right tool? SQL is already a DSL and I can't see how creating another language isn't just adding more overhead without actually getting you to usable objects.

The only reason objects are the "wrong" abstraction is because they don't match relational models exactly. That impedance mismatch is the entire reason for the object-to-relational mapping, otherwise you can use things like document-stores and just serialize your objects directly.

Re: An Unlikely Database Migration

#148

Post by cbushko »

bradfitz wrote: post #141
cbushko wrote: post #131

Earlier quoted context omitted.

Hopefully tailscale gets to the size where kubernetes is worth it. It's a complex thing to run and understand but in the end I think it is worth it. It has certainly made my day to day life a lot easier and allowed our tiny team to build out a solid platform. It has greatly reduced the amount of time that our developers need to get a service up and running our new features out.

We have a lot of Kubernetes experience on the team. Multiple of us run Kubernetes clusters in our home labs (mine: https://github.com/bradfitz/homelab ), and one of us used to be on the Google GKE team as an SRE, and is the author of https://metallb.universe.tf/ (which multiple of us also use). Us _not_ using Kubernetes isn't because we don't know how to use it. It's because we _do_ know how to use it and when _not_…

Haha, it sounds like you have it covered. Even more so if you were to run on GKE (which I use and adore).

When not to use it is a tough question. If I was ever in charge of a company, kubernetes would be the only way of running my product that I would consider. I am a fan of kubernetes as I use it every day but I have also been on the other side of the fence. I have run production systems on bare metal, VMs, EC2 instances, etc. The operational burden of anything non-kube is too much and takes time away from solving big problems such as stability, scaling, deploy, monitoring and more. The solutions to the problems become standard, boring and consistent.

I say the above as someone that spent over a year migrating an entire platform/product from ECS to GKE. It is not perfect but so many silly day to day interruptions have been eliminated. Retired and broken instances are a thing of a past. Scaling is easy. Stability is easier.

Side effects of the move are that our Ops team is 1/2 the size it was a year ago (attrition/covid), we are running 3 times the number of product stacks for 1/3rd the cost. I should really blog about that one!

Re: An Unlikely Database Migration

#150

Post by filleduchaos »

MrStonedOne wrote: post #140
manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

Modern ORMs don't let you hand craft sql, shunting it away in some scary extension that you then have to fight when management on using. A ORM that worked on the principle of insert query text of any complexity receive object as the primary usecase, not the "nonstandard and non-idiomatic usecase" would be the only way to ease the concerns of dba's who code like me. Its the same pitfall of api clients. Why would I tak…

> Modern ORMs don't let you hand craft sql

What "modern ORM" are you using?

Sure I don't write PHP, but the ORMs I've used in Java/Kotlin, Ruby, JavaScript and lately Rust and Crystal have all been more than fine with me writing raw SQL.

Post reply on HN