A database for 2022 • Hacker News Board

Re: A database for 2022

#261

Post by tptacek » Sat, Apr 02, 2022, 4:41 PM UTC

jeffffff wrote: post #259

yawaramin wrote: post #258

Earlier quoted context omitted.

Sounds like you need to do a lot more research before commenting on this? It's pretty easy to find in the Litestream docs that it replicates to S3, Google Cloud Storage, Azure, and other options even including SFTP. In fact by off-loading the storage integration details to Litestream, the Tailscale people now get seamless storage vendor independence almost for free. This is what a really smart and future-proof soluti…

or they could have used mysql or postgres hosted by aws, gcp, azure, etc? i would put money on this not being the last time they change databases

Again: this is an infrastructure component, running on many machines internally. It's not a Rails app. We use SQLite in very similar circumstances across hundreds of machines. Having all those machines schlepping all their reads back to RDS would not only be untenably slow, but it would also make the whole system less stable. I don't think you've really thought through the design at all, and you should before making comments like these.

I'd put money on this not being the last database change too! But not for the same reasons you would.

Re: A database for 2022

#262

Post by tptacek » Sat, Apr 02, 2022, 4:42 PM UTC

jeffffff wrote: post #260

mrkurt wrote: post #257

Earlier quoted context omitted.

The thing about startup decisions is: most of them are "wrong". Or they start right and become wrong later. Successful startups aren't successful in spite of their wrong decisions, they're successful because they can change them very quickly (then write a blog post about it and get more customers). There's also a strong correlation between people who look at things from weird angles and also build good products. Why…

it's not cargo culting if it works, and they even say in the article that mysql or postgres would've worked maybe next they will stop cargo culting operating systems and switch to SerenityOS?

They're talking about running many instances of MySQL locally, not hooking all their systems up to RDS.

Re: A database for 2022

#263

Post by powera » Sat, Apr 02, 2022, 4:47 PM UTC

jrockway wrote: post #162

tptacek wrote: post #139

Earlier quoted context omitted.

Another comment on this thread tried to dunk on them, saying that they'd done this so many times before (JSON to etcd, etcd to sqlite) that they must be getting really good at it. That comment struck me as so close to understanding what they're doing right here, it was painful.

Yeah. Maybe people are overestimating how expensive this kind of change is. We can all imagine being part of a larger org, where one day an All Hands appears on your calendar, and some exec you've never heard of proclaims the beginning of a 2 year multi-team database migration project. So now instead of making nifty software, you are just in 40 hours of meetings a week fighting over small details on this migration no…

Yup.

When Tailscale migrates databases, it is a few engineers working for a few weeks solving problems that already exist.

When a company with 50k employees does it, it takes two years and only makes things worse.

Re: A database for 2022

#264

Post by klysm » Sat, Apr 02, 2022, 4:51 PM UTC

unfocussed_mike wrote: post #218

alophawen wrote: post #209

Not commenting on tailscale, but for the state of databases. Sometimes boring is the right choice. PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted. If you do this for the money, investing in tried and true (but boring) software should be the default solution. I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using P…

Yep. IMHO MySQL becoming boring (fast, pretty reliable, non-compliant quirks being ironed out, decent backup tool availability, credible admin tools, JSON and CTE support) is one of the most important long term trends on the internet. Along with the parallel evolution of PHP.

I strongly dislike the combination of php and MySQL, from the (recent) experiences I’ve had with it. I don’t see a good reason to pick that as a greenfield stack, but I could be convinced otherwise.

Re: A database for 2022

#265

Post by jeffffff » Sat, Apr 02, 2022, 4:54 PM UTC

tptacek wrote: post #158

jeffffff wrote: post #156

Earlier quoted context omitted.

i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their besp…

You should start by learning more about the product, and then tell them they should use Aurora for all their backing store.

No post body was provided.

Re: A database for 2022

#266

Post by blizz017 » Sat, Apr 02, 2022, 5:00 PM UTC

epolanski wrote: post #195

jeffffff wrote: post #124

Earlier quoted context omitted.

if they're this talented, is their time really best spent on dba work rather than improving the product?

Some examples: they could get rid of that pointless bootstrap on their website, they are shoving almost 1.5 MBs for a single font alone on their main page and their HTML semantics are nowhere to be found. This will all impact their bounce rate, accessibility and SEO. I just don't believe the tale of "such skilled engineering teams" which don't show that in their products but blogposts.

What if I told you that the product engineering team is almost never the same team maintaining the website; hell it’s likely the website is contracted out and maintained by the marketing team.

Re: A database for 2022

#267

Post by kall » Sat, Apr 02, 2022, 5:06 PM UTC

At first I was also a little confused by this architecture. I think that was because I was missing a clear picture of what they want to do. I think it is:

- one leader that can write to the db

- a ton of other hosts that can read from the "streamed" db and want to do so often/fast

- hosts send their writes (if they have any?) to the leader via some api call (?)

Is that right? If so, the solution makes a ton of sense. "Just use pg/MySQL" in that case would mean hundreds (thousands?) of read replicas, which doesn't sound fun, or every read over the network.

Re: A database for 2022

#268

Post by 0des » Sat, Apr 02, 2022, 5:12 PM UTC

ignoramous wrote: post #198

techdragon wrote: post #187

Earlier quoted context omitted.

This and the “no Wi-Fi less space than a Nomad” comments on slash dot about the iPod are my favourite examples of “you don’t get consumers” comments.

to quote u/InGoodFaith (news.ycombinator.com/item?id=29555814): > Wouldn't also be HN without the misinterpretation of BrandonM's response. Here is dang's comment about that topic/meme: https://news.ycombinator.com/item?id=29178442

As someone who was there to witness it when it happened, that is not the impression I got at all, and I think Dang may have taken the more humble interpretation in that remark.

Re: A database for 2022

#269

Post by belter » Sat, Apr 02, 2022, 5:12 PM UTC

emteycz wrote: post #236

belter wrote: post #225

Earlier quoted context omitted.

That comment is unfairly judged, taking into account how much they used AWS instead of their own solution. "...Half-a-billion people stored files on Dropbox. Well, sort of. Really, the files were in Amazon’s cloud. .." [1] [1] "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire" https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...

Why can't a company use cloud? It's not like there's Amazonbox client I could install now and get the same service. Dropbox never was about the storage itself, IMHO - it was about the ease of usage. There were enough options to get storage - cloud or non-cloud - when it came out, but none of them had the easy folder-based sync and sharing (public folder) of Dropbox.

It's not if the company uses cloud or not. It's if they are solving the hard problems or instead, outsourcing the technical challenges to their cloud provider while charging the user a non justifiable premium for the intuitive user interface :-) So it's about the eventual value add and the business case.

The challenging part is to provide the back-end. Get massive storage, with proper checksum integrity, at a cost effective value and with flawless security.

A single command (and minor variations around it):

aws s3 sync s3://

get's you all that. It can be easily mapped to an intuitive user interface by a software development boutique in weeks.

And because that is the difficult part, it seems for the first 8 years of their history that was solved for them by AWS, according to their own reported history.

Sharing a folder or file from Dropbox is no more intuitive than using the S3 console. Looking at the very colorful history of security issues around Dropbox [1] it's certainly not safer.

So why would a user pay more for storage costs than with a cloud provider, with notoriously less security? Does a tiny layer on top one or more cloud providers justify the add-on value for the user or the current valuation of 9 Billion dollars?

[1] "Criticism of Dropbox": https://en.wikipedia.org/wiki/Criticism_of_Dropbox

Re: A database for 2022

#270

Post by petethepig » Sat, Apr 02, 2022, 5:13 PM UTC

I set up a similar system in the past with redis replicas — it was about 200 MBs of data that I needed to have on quick access to on ~ 300 servers around the world — it was easy to set up and worked really well.