Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A database for 2022

tailscale.com

Post reply on HN
Print view View original

161–170 of 336 posts

Re: A database for 2022

#161

Post by AnimalMuppet »

billywhizz wrote: post #150
AnimalMuppet wrote: post #115

Earlier quoted context omitted.

Seconded. It's a logical fallacy because they are using the year as proof that we should adopt their policy. Except the year only proves what year it is - nothing else.

oh boy. is nobody allowed to make a little joke anymore. you should take yourself a lot less seriously my dude.

First: I am not "my dude" to you.

And, completely apart from the database, it's a really valid point - not a joke at all.

When someone says, "It's 2022. Why are we still arguing about X?" or some such, it means that they haven't been able to persuade the rest of the world about X on the merits of their argument, and so they want to short-circuit the argument by appearing "modern" or "up to date" or "on the right side of history". They want to win by persuading the other side to give up in embarrassment, not by virtue of the quality of their position. (In their own eyes, their position is probably self-evidently the only right and moral one, but that doesn't tell us much.) It's a way of trying to win by default, and it deserves nothing other than being regarded as "you lose by default".

Re: A database for 2022

#162

Post by jrockway »

tptacek wrote: post #139
jrockway wrote: post #127

Earlier quoted context omitted.

People seem really stressed that they've changed databases twice before. Their service never went down and they didn't stop delivering features while all this was happening, so I don't see the big deal. You pick what works at the time, and incrementally evolve as problems and solutions become apparent. This article could have easily been "why we were down for a day while some auto-vacuum setting was broken." Then eve…

Another comment on this thread tried to dunk on them, saying that they'd done this so many times before (JSON to etcd, etcd to sqlite) that they must be getting really good at it. That comment struck me as so close to understanding what they're doing right here, it was painful.

Yeah. Maybe people are overestimating how expensive this kind of change is. We can all imagine being part of a larger org, where one day an All Hands appears on your calendar, and some exec you've never heard of proclaims the beginning of a 2 year multi-team database migration project. So now instead of making nifty software, you are just in 40 hours of meetings a week fighting over small details on this migration nobody wants to do, and everyone is miserable about it. Features stopped getting released, people leave the company, the weekly status meeting never makes the project seem anywhere near completion.

Say "switch databases on a successful project" and I bet people's minds go there. That is indeed expensive and painful, but if you have a small team, it's just being agile.

There is also the "punching up" / sour grapes aspect that can't be ignored. "I would have picked the right database from the start, but nobody's funding my crazy idea." Life just isn't fair, and HN is where you can vent, I guess.

Re: A database for 2022

#163

Post by statictype »

tptacek wrote: post #137
statictype wrote: post #131

Earlier quoted context omitted.

It would be interesting to known why a standard boring RDS setup wouldn’t solve their problem completely. In fact I would be more interested to understand that than the actual details of sqlite tailing. (I think the reason they gave was vendor lock-in, but apart from that, I didn’t understand why it wouldn’t be adequate)

One obvious reason not to use RDS is wanting disk-local caches of information replicated from a single leader, rather than having every single machine in your fleet calling out to an external service on every read . That's certainly why we're not considering Postgres in our infrastructure, even though managed Postgres is a product we in fact offer. We use Postgres! It's the backing state for our API, and an important…

Makes sense once you think of Tailscale as an infrastructure-level service and not just an app. Thanks

Re: A database for 2022

#164

Post by BonoboIO »

hparadiz wrote: post #130
tptacek wrote: post #87

This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…

+1 on good design and programming. My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack. I k…

But my Kubernetes Cluster on AWS can handles this for only 30.000$ a month

That Comcast prototype sounds amazing. PHP and MySQL can really bring you very far, i use it for nearly everything and it is so incredible easy to develop and “deploy” new versions. I use propel as ORM and it is very close to the DB and I can change schemes fast and just let the autoupdate handle the rest.

Re: A database for 2022

#165

Post by broken8ball »

mathgladiator wrote: post #122

I believe using a large JSON file is not half bad, but you do run into the problem of how do you index and query it in meaningful. I actually ran into this problem when building a game because a document doesn't provide a great model.. The relational model is... JUST... SO... GOOD. And, it is a shame that most of the relational systems are so complicated. A document within Adama ( https://www.adama-platform.com/ ) is…

Really curious…how do you meaningfully overlay any indexes on top of a big-ass JSON file? Technical details are appreciated, and no problem if it’s your secret sauce—just very curious how this is accomplished!

Re: A database for 2022

#166

Post by ignoramous »

jeffffff wrote: post #160
ignoramous wrote: post #133

Earlier quoted context omitted.

> is their time really best spent on dba work... It seems to me that tailscale engs want to avoid DBA work but also not use managed offerings, and so, they're comfortable paying the costs they have to (such as multiple migrations). > ...rather than improving the product? Well, you'd guess they want to be able to continually improve their already credible product too. When TFA points out that zero vendor lock-in and h…

> When TFA points out that zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, I think it is for this reason. if zero vendor lock-in and hassle-free, local end-to-end tests are non-negotiable, why are they using s3? migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted…

> migrating to another s3 compatible backend would be similar in effort to migrating from aurora mysql or postgres to another managed mysql or postgres service or to self-hosted mysql or postgres

You may be right. I have no experience migrating litestream but from the docs (https://litestream.io/guides/) it is literally cp'ing files from S3 to wherever and exec'ing one of these one-liners (of course, the devil is in the details):

   litestream restore -o my.db s3://BUCKETNAME/PATHNAME
   litestream restore -o my.db abs://STORAGEACCOUNT@CONTAINERNAME/PATH
   litestream restore -o my.db gcs://BUCKET/PATH
   litestream restore -o my.db s3://SPACENAME.nyc3.digitaloceanspaces.com/db
   litestream restore -o my.db s3://BUCKETNAME.us-east-1.linodeobjects.com/db
   litestream restore -o my.db sftp://USER:PASSWORD@HOST:PORT/PATH

Re: A database for 2022

#167

Post by ramraj07 »

rablackburn wrote: post #100

I loved this article, despite the negativity. A discussion that keeps recurring on HN is how you can get a lot done with simple, boring solutions. This is a perfect example of that in practice. What’s the simplest thing that’ll work? Do that, monitor the solution, and when you start hitting a limitation reevaluate what the next appropriate solution is. There are plenty of comments here deriding this as insanity when…

The question is, is this the simple solution? Perhaps I and others have been spoiled by cloud offerings but having set up RDS once and never having to worry much about the db architecture after that for years as we scale has maybe spoiled us.

Re: A database for 2022

#168

Post by mbreese »

jimbokun wrote: post #157
tptacek wrote: post #135

Earlier quoted context omitted.

They don't, but there are other "finesse" dishes that people do in fact go to restaurants for that are notable for their simplicity, and the fact that you can only pull them off if you have flawless technique. Japanese, in particular, is notorious for this.

There’s a manga about a food critic (Oishinbou [1]), where he takes another food critic to a hole in the wall Tokyo restaurant as an example of elite cooking. The chef-owner brings him his meal. Which is just a piece of cooked fish, rice, and miso soup. The critic is first almost insulted by such a simple meal. But then realizes the fish, rice and soup are prepared to perfection, and is amazed at the chef’s skill. [1…

Which is basically also the ending of the Pixar film Ratatouille. The protagonist wows a critic with a simple dish prepared to perfection. It’s a common trope, but also true. “Simple” food prepared well can be much better than a complex dish with many flavors.

Similarly, I’ve also heard (probably from a show like Top Chef, so YMMV) things like a salad or soup are also quick ways to differentiate chefs. In this case, it’s less about technique and more about identifying and mixing flavors.

It works because there is no place to hide. You either have good flavor or not.

Re: A database for 2022

#169

Post by Winsaucerer »

mrkurt wrote: post #79
KwisaksHaderach wrote: post #13

The obvious candidates were MySQL (or one of its renamed variants given who bought it) or PostgreSQL, but several of us on the team have operational experience running these databases and didn’t enjoy the prospect of wrestling with the ops overhead of making live replication work and behave well. Other databases like CockroachDB looked very tempting, but we had zero experience with it. And we didn’t want to lock ours…

Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.

I'm having a really difficult time migrating my managed postgresql instance in Google Cloud just to get it to a newer version. I'm down to my last option: shut down everything, export, re-import somewhere else again.

I did briefly have logical replication working, and so had my near zero downtime solution in hand. But the replication broke after a few days because I delayed cutting over, and then I couldn't get it going again. I couldn't find out how to debug this either.

The lesson I've learned here is that I'll be avoiding managed DB products whenever I can. It takes away control that turns out I sometimes really need.

Re: A database for 2022

#170

Post by ransom1538 »

tptacek wrote: post #87

This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…

One of the largest forums online in 2003 was was ezboard. A billion page views per month. Back then - insane - well that was beyond insane. I started work there as a young idiot. "What db you guys using, mysql?...." long pause... "We use a file system." I dug in. Yep. A fucking file system. Files allowed: simple backups, impossible speed, obvious cache systems, partitioning.

Later on i met the roomate of the author of memcache. Few hundred lines of C. Completely changed the game how caching systems worked. Years later a super senior FB engineer explained to me over a beer - "we basically use mysql as a file store, everything else is in memcache." Good ideas are stupid simple.

Post reply on HN