Live data from Hacker News

Spending $5k to learn how database indexes work

briananglin.me

71–80 of 209 posts

Re: Spending $5k to learn how database indexes work

#71
post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

Seriously. Like, every junior dev has to learn DB indexing basics sometime, and apparently the other of this blog post just did. But really can’t understand why this article is getting voted to the top of HN.

Re: Spending $5k to learn how database indexes work

#72
post #71
post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

Seriously. Like, every junior dev has to learn DB indexing basics sometime, and apparently the other of this blog post just did. But really can’t understand why this article is getting voted to the top of HN.

What I gained from the article wasn't that the dev was unaware of indices, it's that he didn't realise indices were missing due to how planetscale's database disallows foreign keys.

I never worked with a database that doesn't have foreign keys and it's not unthinkable to forget when you do for the first time, that foreign keys were what created indexes for you automatically.

A little bit of planning could have prevented that though :/

Re: Spending $5k to learn how database indexes work

#73
post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.

Yes we really should not accept this. The ability to impose limits on spending is key to control an enterprise. Whole security certification guacamole is based on having established controls. But where the bit hits the fan control is absent.

Re: Spending $5k to learn how database indexes work

#74
As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences.

In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing.

I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and slower-scaling (unless explicitly overidden/allowed, of course).

If I accidently push very non-performant code, I kind of expect my service to get less performant, quickly realize the problem, and fix it. I don't expect a service to seemingly-magically detect my poor code, increase my bill by a couple orders-of-magnitude, and only alert me hours (if not days) later.

Re: Spending $5k to learn how database indexes work

#75
post #25

Earlier quoted context omitted.

Thanks Sam! As mentioned in the post, the PlantScale team was quick to credit our account for the overages and help us figure out what was going on. I'm personally super bullish on PlanetScale! With any new product there will be tradeoffs and rough edges but the positives, like easy migrations and database branches have definitely outweighed any difficulties.

Kudos for being open about your mistakes. Could you share a little bit about what your thought process was in general when picking a database technology? You call out "easy migrations and database branches" outweighing other quirks, so some pros and cons weighing must've happened :) Is it easy, for example, to test things in your dev environment with realistic amounts of data, and to get an understanding of how the q…

Thanks! The decision making process was pretty unsophisticated tbh. Basically I spent the last couple years working at GitHub as a security engineer and had been pretty comfortable with MySQL so wanted to stick with that. I had heard from our database team how annoying migrations were and I had previously locked a postgres database multiple times in production trying to deploy a migration, so MySQL + safe migrations + some of the best engineers I've worked with pouring all their time into PlanetScale, made a ton of sense. So basically a combination of proven underlying tech + believing in the team.

The migration workflow is really cool, basically when we create a PR we branch our production database and apply any migrations that are included in that PR and then that branch is used in our per-PR review environment. (Just Heroku Review Apps), then when we merge the PR, we also merge the deploy request in PlanetScale. Database branching is a super powerful concept once you've leaned into it.

We don't really do any sort of load testing in a dev environment. We have one customer who is also a co-founder of our company so we just deploy whatever changes we're unsure about (after automated testing) to his application and see what happens. If anything looks off in Grafana we'll make take a look but it's usually "good enough" or "totally broken", very rarely do we take time to make something 25% better if it already works. The time to fix vs speed of shipping features tradeoff doesn't make sense for us.

In this specific case, the query was taking place in a background job so 10ms - 500ms didn't really matter to us so we didn't really measure the timing, if we had we may have noticed it was slow but kinda a testament to PlanetScale that we didn't even notice ;)

Re: Spending $5k to learn how database indexes work

#76

I'm just leaving this here: https://www.hetzner.com/dedicated-rootserver/ax161/configura... Draw that nice red slide all the way to the right. No, it's not storage. Yeah, it's actually affordable. Yeah, that was a sexual sound you just made. You do have to be prepared to know some basic sysadmin, or pay somebody to do it for you. My newest server has about 60 cores and half a tera of ram. Surprisingly, it's not uber…

What are you using this for?

Re: Spending $5k to learn how database indexes work

#77

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

Haha yep, I was like wait I'm used to getting feedback from the system telling me I messed up and this I barley noticed. PlanetScale has Query Statistics that are really useful for spotting slow queries but don't expose the "rows read" so you can't really tie this view back to billing. I think they're aware of this though and might expose that information.

Re: Spending $5k to learn how database indexes work

#78
post #17

Don’t use DB providers that charge for rows/data scanned. Use Amazon RDS or Google Cloud SQL or just install it yourself on a VM. Pay for CPU, memory, and storage instead.

Rent metal and run your own MySQL/Postgres/...

One insert every 3 seconds. Could run that off a 10 year old laptop.

Re: Spending $5k to learn how database indexes work

#79

I've seen things you people wouldn't believe. Millions burnt on consultants and licensing Oracle. I watched C series startups throwing it all away in a move to NoSQL. All those Amazon RDS fees will be lost in time.

> I watched C series startups throwing it all away in a move to NoSQL

have you forgot that MongoDB is web scale?

https://www.youtube.com/watch?v=b2F-DItXtZs

Re: Spending $5k to learn how database indexes work

#80
post #14

I feel like indexes are a pretty fundamental type of DB knowledge. In fact I'd say it's table stakes knowledge you should have if you're working with them. Further more, knowing that ForeignKeys typically apply an index to that column is also in my head basic knowledge. I'm sorry you got burnt, and congrats on learning a lesson, but you could have gotten the same knowledge by ever googling MySql ForeignKeys and saved…

Good for you. But I think you're being uncharitable by failing to distinguish between "concept I didn't understand" and "thing I forgot to consider until I saw the problem it caused". The title also suggests the former, but I think the author is being a bit humble by underplaying his existing knowledge. Likely he actually did know what indexes are before; if you asked him to detail how MySQL foreign keys work he might have even remembered to say they add an implicit index. But it's super easy to miss that you're depending on a side effect like that until you see the slow query (or, in this case, high bill).

When you're programming, how many compiler errors do you see a day? (For me, easily dozens, likely hundreds.) Do you think each one indicates a serious gap in your knowledge?

Along these lines: imposter syndrome is a common problem in our industry. One way it can manifest is junior engineers can thinking they're bad programmers when they repeatedly see walls of compiler errors. I think it'd help a lot to show them a ballpark of how often senior engineers see the same thing. [1] I know that when I'm actively writing new code (especially in languages that deliberately produce errors at compile time rather than runtime), I see dozens and dozens of errors during a normal day. I don't think this is a sign I'm a bad programmer. I think it just means I'm moving fast and trusting the compiler to point out the problems it can find rather than wasting time and headspace on finding them myself. I pay more attention to potential errors that I know won't get caught automatically and particularly to ones that can have serious consequences.

I think the most important thing the author learned is that failing to add an index can cost this much money before you notice.

Ideally the author and/or the vendor will also brainstorm ways to make these errors obvious before the high bill. Load testing with realistic data is one way (though people talk about load testing a lot more than they actually do it). Another would be watching for abrupt changes in the operations the billing is based on.

[1] This is something I wish I'd done while at Google. They have the raw data for this with their cloud-based work trees (with FUSE) and cloud-based builds. I think the hardest part would be to classify when someone is actively developing new code, but it seems doable.

Post reply on HN