Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

161–170 of 190 posts

Re: An Unlikely Database Migration

#161

Post by sombremesa »

cavisne wrote: post #132

Honestly this feels like engineers that spent too long in FANG and get completely burnt out on dealing with SRE and HA requirements... so decide to built a setup so prone to 2AM pages even a PHP webshop would frown at it. Curiously though its a pattern I've seen twice in the last 12 months, there was that guide on the good bits of AWS that also recommended starting with a single host with everything running on it. Ma…

For me the most shocking part was that this company is spending time and money on overengineering solutions when they know that a JSON file got them that far — SQLite would be a perfectly fine improvement to get themselves further!

I had no idea companies of this size had engineers with that much free time on their hands.

Re: An Unlikely Database Migration

#162

Post by perlgeek »

jeff-davis wrote: post #7

This post illustrates the difference between persistence and a database. If you are expecting to simply persist one instance of one application's state across different runs and failures, a database can be frustrating. But if you want to manage your data across different versions of an app, different apps accessing the same data, or concurrent access, then a database will save you a lot of headaches. The trick is kno…

I'd say the universal query capability is the killer feature of SQL.

In the OP they spent two weeks designing and implementing transaction-save indexes -- something that all major SQL RDBMS (and even many NoSQL solutions) have out of the box.

Re: An Unlikely Database Migration

#163

Post by jsumrall »

nine_k wrote: post #152
manigandham wrote: post #147

Earlier quoted context omitted.

Do you have an example of the right tool? SQL is already a DSL and I can't see how creating another language isn't just adding more overhead without actually getting you to usable objects. The only reason objects are the "wrong" abstraction is because they don't match relational models exactly. That impedance mismatch is the entire reason for the object-to-relational mapping, otherwise you can use things like documen…

SQLAlchemy gives a nice composable DSL in Python; it has an ORM layer on top of it, too, but it's optional. JOOQ allows for composable typesafe SQL in Java.

+1 for jOOQ!

Re: An Unlikely Database Migration

#164

Post by ignoramous »

bradfitz wrote: post #47
mwcampbell wrote: post #44

Earlier quoted context omitted.

> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?

> What if you used one of the managed RDBMS services offered by the big cloud providers? We could (and likely would, despite the costs) but that doesn't address our testing requirements. The control plane is on AWS. We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.

> ...but the most important bit is on AWS.

Curious: Was running DynamoDB with DAX (DynamoDB Accelerator) in front ever in contention? If not, is it due to vendor lock-in (for example, not being able to migrate out) or because tailscale doesn't feel the need to use managed offerings especially for core infrastructure?

Re: An Unlikely Database Migration

#165

Post by masklinn »

AlfeG wrote: post #160
miki123211 wrote: post #99

I use the same system (a JSON file protected with a mutex) for an internal tool I wrote, and it works great. For us, file size or request count is not a concern, it's serving a couple (internal) users per minute at peak loads, the JSON is about 150 kb after half a year, and old data could easily be deleted/archived if need be. This tool needs to insert data in the middle of (pretty short) lists, using a pretty compli…

That's good. But single file could break on powerloss. I use sqllite. It's quite easy to use, not a single line though.

Their point about schema migration is completely true though. An SQLite db is extremely stateful, and querying that state in order to apply schema migrations (for both the data schema and the indexes) is bothersome.

Re: An Unlikely Database Migration

#166

Post by Cthulhu_ »

manigandham wrote: post #137

> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…

> Modern ORMs are fast, efficient, and very productive.

The author seems to be using Go, which honestly could use work in that area. gorm is the biggest / most popular ORM out there, but it looks like a one-person project, the author seems well worn-out already, and it kinda falls apart when you work with a data model more than one level deep.

Plus broadly speaking, there seems to be a bit of an aversion to using libraries in the Go community.

Re: An Unlikely Database Migration

#167

Post by Cthulhu_ »

bob1029 wrote: post #5

I do like putting .json files on disk when it makes sense, as this is a one-liner to serialize both ways in .NET/C#. But, once you hit that wall of wanting to select subsets of data because the total dataset got larger than your CPU cache (or some other step-wise NUMA constraint)... It's time for a little bit more structure. I would have just gone with SQLite to start. If I am not serializing a singleton out to disk,…

I've seen the same when at one point we decided to just store most data in a JSON blob in the database, since "we will only read and write by ID anyway". Until we didn't, sigh. At least Postgres had JSON primitives for basic querying.

The real problem with that project was of course trying to set up a microservices architecture where it wasn't necessary yet and nobody had the right level of experience and critical thinking to determine where to separate the services.

Re: An Unlikely Database Migration

#169

Post by psankar »

jeff-davis wrote: post #7

This post illustrates the difference between persistence and a database. If you are expecting to simply persist one instance of one application's state across different runs and failures, a database can be frustrating. But if you want to manage your data across different versions of an app, different apps accessing the same data, or concurrent access, then a database will save you a lot of headaches. The trick is kno…

This comment helped me understand the problem and the solution better (along with a few followup tweets by the tailscale engineers). Thanks.

Re: An Unlikely Database Migration

#170

Post by npiit »

enneff wrote: post #155
npiit wrote: post #9

Earlier quoted context omitted.

I understand. But given the product features compared to the rest of the industry, Tailscale brings no real value compared to Zero-tier hen it comes to meshes, it's not zerotrust like Cloudflare, Twingate, and many others, it claims to be open source while only the client is and it cannot be used without their closed source control plane where most of the feature are behind paywall, it's way more expensive than reput…

> Tailscale brings no real value compared to Zero-tier This article has nothing to do with Tailscale the product and everything to do with the team's unconventional approach to engineering. That's what HN is interested in and why the post is being upvoted.

There is nothing unconventional in moving from SQL to key-value distributed database. And if it was any other company that submitted this very same post here we wouldn't be talking here right now as it would have never gotten a single upvote. The posts of this company almost always come with their upvotes right after submission (by others) and the founders were surprisingly replying minutes after submission. This is systematic behavior.
Post reply on HN