Live data from Hacker News

Spending $5k to learn how database indexes work

briananglin.me

151–160 of 209 posts

Re: Spending $5k to learn how database indexes work

#151

Earlier quoted context omitted.

Like tears in the rain

That last metaphore was added by the actor himself...the director asked him to have some human showing in him, and now it's history.

Actually, the whole part was added by the actor, not just the last metaphor.

Re: Spending $5k to learn how database indexes work

#152

Earlier quoted context omitted.

That last metaphore was added by the actor himself...the director asked him to have some human showing in him, and now it's history.

Actually, the whole part was added by the actor, not just the last metaphor.

I think we're both right :)

https://en.wikipedia.org/wiki/Tears_in_rain_monologue?wprov=...

Re: Spending $5k to learn how database indexes work

#153

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

> In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing.

Oh, that would be actually quite useful for learning things if the bill would tell you that it got so high because you stupid dump-ass didn't use DB indices properly.

I'm every time shocked how many people using DBs don't know about indices! Those people should pay such a bill once. They would never ever again "forget" about DB indices I guess.

Of course I'm joking to some extend. But only to some extend…

Re: Spending $5k to learn how database indexes work

#154

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

There's no free lunch. Cloud services trade performance woes for budget surprises. This may be preferable in some cases but the tradeoff should be recognised.

> This may be preferable in some cases but the tradeoff should be recognised.

It's not a "tradeoff", it's a product feature.

Re: Spending $5k to learn how database indexes work

#155

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

There's no free lunch. Cloud services trade performance woes for budget surprises. This may be preferable in some cases but the tradeoff should be recognised.

There's plenty of space in the middle though, no? Bank accounts cut you off if you hit a zero balance, or they can execute your transactions and charge you a fee. Why can't these services implement throttling or even halting if the charges hit a certain ceiling?

Re: Spending $5k to learn how database indexes work

#156

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

The question that follows would be: how do you know what was intended to be less performant versus optimized on-demand? The intentions can be easily inferred when the query at hand was a simple join, and to no surprise, many cloud database offerings _do_ provide optimization automation (Azure SQL will for example even automatically add obvious indexes if you let it). But what if the query did need to scan all the rows in a join, but was only a one-off, and you didn’t want to pay the continued perf and storage costs of maintaining an index? The cloud provider can’t know that, and even with proactive measures (“make it slower” can’t work because speed is part of the product design, and budget controls can only go so far before it impacts your own customers) there’s only so much that can be done. The choice of infinity scale tools comes with infinity scale costs, and so there’s a responsibility that engineers using these tools need to understand what they’re accepting with that choice.

Re: Spending $5k to learn how database indexes work

#157
post #143

Earlier quoted context omitted.

If those million items need to be read from different servers -- as they might well, in a distributed database -- it's definitely not just a few ms of computing power. For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb.

> For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb. Is that the same counting method as PlanetScale's "row read"? That is, `select title from posts order by title limit 10` on a table with 10 million rows and no index on `title` would cost $1.25 per query?

DynamoDB doesn't have SQL queries; but yes, if you're performing an operation which reads 10 million items from ddb it will be absurdly expensive. It will also take an absurdly long time; by default ddb is limited to 40k read request units (= 80k eventually consistent reads of up to 1 kB) per second. Being so slow would probably make users realize they're doing something wrong.

Re: Spending $5k to learn how database indexes work

#158

Earlier quoted context omitted.

There's no free lunch. Cloud services trade performance woes for budget surprises. This may be preferable in some cases but the tradeoff should be recognised.

There's plenty of space in the middle though, no? Bank accounts cut you off if you hit a zero balance, or they can execute your transactions and charge you a fee. Why can't these services implement throttling or even halting if the charges hit a certain ceiling?

In some cases the query might have finished before the data hits the billing system.

Re: Spending $5k to learn how database indexes work

#159

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

There's no free lunch. Cloud services trade performance woes for budget surprises. This may be preferable in some cases but the tradeoff should be recognised.

> Cloud services trade performance woes for budget surprises.

I'm not sure why you think this is a trade-off. In general cloud services automate operations. Whether they are faster is unrelated. Many are not--services that use object storage for backing storage can be orders of magnitude slower than equivalent software using nVME SSD.

Re: Spending $5k to learn how database indexes work

#160
post #86

Earlier quoted context omitted.

No you've missed my point, the author seemingly didn't know that ForeignKeys applied indexes by default in MySql. It's not "Concept I didn't understand", clearly they're capable of understanding because they did after they ran into the issue. It's about not having had basic knowledge to begin with. But he didn't see compiler errors, he caused monetary cost to his employer. When I deploy something that unintentionally…

> When I deploy something that unintentionally causes a large monetary bill to my employer, then yes I do believe that indicates a gap in knowledge so I don't in anyway believe I'm being uncharitable. It depends, if you've been given a loaded footgun it's not entirely your fault when it inevitably goes off. Let's go back to your "compiler errors" scenario, and let's say someone decided that the company should be usin…

>The problem is that there is a DB that charges insane amounts of money per row processed with no upper limit and that someone actually thought it was a good idea to use it.

That's it in a nutshell. Usually you have an upper bound on compute, memory, disk space or some other resource for a specific price. When you hit those limits, you find performance issues and at that point you can choose to try optimizing your code or database, then decide whether you need to upgrade resources at cost.

I really don't understand this model that charges for rows read or, worse, "inspected". What's the upside of that model versus more typical pricing schemes, and how is it manageable/predictable from a budget perspective? With or without the indexing problem here, you'd really have to know your user behavior, then translate that to DB read counts by your app. And, while devs should all be optimizing code as much as reasonable, something as specific as minimizing DB reads seems an odd constraint to place on software.

I'm guessing there must be some use case I'm missing; else I don't know why this pricing scheme is even a thing.

Post reply on HN