Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spending $5k to learn how database indexes work

briananglin.me

Post reply on HN
Print view View original

131–140 of 209 posts

Re: Spending $5k to learn how database indexes work

#131

Post by ampersandy »

Grimm1 wrote: post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page.

Because it's a well written, humble account of learning from a mistake then using it as an opportunity to teach others to help them avoid the same mistake.

If anyone leads a team, I hope they might learn from this approach, rather than just bashing on people and implying they don't deserve any attention because they made a mistake a more experienced developer might have dodged.

Re: Spending $5k to learn how database indexes work

#132

Post by williamdclt »

Nextgrid wrote: post #18
Grimm1 wrote: post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.

> But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.

No. Nobody finds that "normal", that's just untrue. It's even the _whole_ subject of this blogpost: the bill was not normal.

I don't disagree that some people are overrelying on cloud services, but that didn't become normality, it's still a beginner's mistake

Re: Spending $5k to learn how database indexes work

#133

Post by williamdclt »

mike_hock wrote: post #78
dreyfan wrote: post #17

Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.

Rent metal and run your own MySQL/Postgres/... One insert every 3 seconds. Could run that off a 10 year old laptop.

They clearly don't have the skills for that. And at one insert every 3s, a managed service like RDS won't really cost much.

Re: Spending $5k to learn how database indexes work

#135

Post by xorcist »

Buttons840 wrote: post #59
Nextgrid wrote: post #42

Earlier quoted context omitted.

I have no problems with a developer doing a 5k oopsie with things like card processing or an area that has a legitimate potential for direct monetary losses (such as payment processing where a bug could allow customers to order goods without actually paying). I have a problem with whoever looked at and said "nah, let's go with a cloud-based solution that charges per-query and gives us an essentially infinite financia…

It's not so clear cut. What's the cost of losing the entire on-prem database? Do you trust a company who hired a developer who didn't know about indexes to hire a rock solid DBA? And how much does that DBA cost?

This is not a matter of hiring an elite DBA. This is a matter of reading the manual. Both indices and backups are right there in chapter 7 and 8 respectively. But that's something worth doing irrespectively if you are running your own database or using someone else's as-a-service.

Re: Spending $5k to learn how database indexes work

#136

Post by williamdclt »

pjscott wrote: post #84
Grimm1 wrote: post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

One of the best database habits I've ever developed is to run EXPLAIN on every query that I expect to run repeatedly, then sanity-check the output. It's very little effort, and has prevented so much hassle.

If we weren't using underwhelming ORM DSLs, I'd love to use/write a github bot that automatically runs EXPLAIN ANALYZE on queries updated in a PR and post the query plan!

Re: Spending $5k to learn how database indexes work

#138

Post by lmilcin »

nrmitchi wrote: post #74

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

Why not write a simple service that tracks various stats (like number of users, requests, etc.) as well as billed costs over time?

You could then get various interesting stats in real time as well as some pretty useful alerting.

Re: Spending $5k to learn how database indexes work

#139

Post by hodgesrm »

My company runs a cloud service for ClickHouse. We've spent a lot of time thinking about pricing. In the end we arrived at (VMs + allocated storage) * management uplift + support fee.

It's not a newfangled serverless pricing model, but it's something I can reason about as a multi-decade developer of database apps. I feel comfortable that our users--mostly devs--feel the same way. We work to help people optimize the compute and storage down to the lowest levels that meet their SLAs. The most important property of the model is that costs are capped.

One of the things that I hear a lot from users of products like BigQuery is that they get nailed on consumption costs that they can't relate in a meaningful way to application behavior. There's a lot of innovation around SaaS pricing for data services but I'm still not convinced that the more abstract models really help users. We ourselves get nailed by "weird shit" expenses like use cases that hammer Zookeeper in bad ways across availability zones. We eat them because we don't think users should need to understand internals to figure out costs. The best SaaS services abstract away operational details and have a simple billing model that doesn't break when something unexpected happens on your apps.

Would love to hear alternative view points. It's not an easy problem.

Re: Spending $5k to learn how database indexes work

#140

Post by Grimm1 »

ampersandy wrote: post #131
Grimm1 wrote: post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. Because it's a well written, humble account of learning from a mistake then using it as an opportunity to teach others to help them avoid the same mistake. If anyone leads a team, I hope they might learn from this approach, rather than just bashing on people and implying they don't deserve any attention because they…

Personally I find it trite but whatever floats your boat.
Post reply on HN