Live data from Hacker News

Spending $5k to learn how database indexes work

briananglin.me

171–180 of 209 posts

Re: Spending $5k to learn how database indexes work

#171

Earlier quoted context omitted.

There's plenty of space in the middle though, no? Bank accounts cut you off if you hit a zero balance, or they can execute your transactions and charge you a fee. Why can't these services implement throttling or even halting if the charges hit a certain ceiling?

In some cases the query might have finished before the data hits the billing system.

That’s not an argument.

As long as query #2 doesn’t run, that’s better then nothing.

Re: Spending $5k to learn how database indexes work

#172

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

It can't be like that. I have discussions with vendors sometimes and the first question I ask - if something lapses and we weren't paying attention - you won't cut our service right? I think too, in most cases, people would rather run over than cut service. Also how would such a system work? Let's say you sign up for some API and what, set your billing limit to 500 requests per day. Let's say you're now hitting fabul…

If you’re expecting something near 500, then why would you set your limit to 500. Set it towards something like 20k at least.

Or obviously if you don’t think this will be a problem, you have control to set it to uncapped.

I don’t understand what argument you have.

Re: Spending $5k to learn how database indexes work

#173

Earlier quoted context omitted.

In some cases the query might have finished before the data hits the billing system.

That’s not an argument. As long as query #2 doesn’t run, that’s better then nothing.

So you want to add an overhead for each query to check an internal LRU cache that then checks the billing system? Just the overhead of hashing the query into some cacheable identifier will hurt performance.

Re: Spending $5k to learn how database indexes work

#174

The real answer here is cost limiting. I don't want my cloud provider to keep working at the cost of an order of magnitude higher bill than I was expecting because of a bug in my code. I want to be able to set a billing limit and have them degrade or stop there service if I exceed the limit. AFAIK AWS doesn't have that. They do have the ability to send me alerts if my bill is unexpectedly high, but they still keep wo…

That's why we use DigitalOcean a lot in Africa. You know upfront how much you will spend.

You can calculate how much RDS is gonna cost you per month beforehand.

In fact, it is slightly cheaper at AWS.

Ondemand PostgreSQL, Single Node, 1vCPU, 1GB MEM, 10GB Storage is $15 at DO

Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $14.29 at AWS (db.t3.micro at us-east-2)

if reserved for 1yr no upfront

Reserved PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $10.57 at AWS (db.t3.micro at us-east-2)

Or you can use ARM and go lower.

Ondemand PostgreSQL, Single Node, 2vCPU, 1GB MEM, 10GB Storage is $12.83 at AWS (db.t4g.micro at us-east-2)

Re: Spending $5k to learn how database indexes work

#175
TLDR: author forgot to create indexes in cloud-based MySQL database and paid too much for the queries which were run as full-table scans.

Interestingly enough, some DBs (like Cassandra) would refuse scan-type queries unless specifically asked to. I wonder if cloud-based DBs which charge per row inspected could have such mode... Though of course it's their incentive not to.

Re: Spending $5k to learn how database indexes work

#176

I'm not a DB expert, but "750k users in a month." doesn't sound like a quantity that you'd need to use some kind of fancy special tooling for.

I was thinking the same. It sound like this could be done on a private virtual server for less than $100/month.

Re: Spending $5k to learn how database indexes work

#177

Earlier quoted context omitted.

The question that follows would be: how do you know what was intended to be less performant versus optimized on-demand? The intentions can be easily inferred when the query at hand was a simple join, and to no surprise, many cloud database offerings _do_ provide optimization automation (Azure SQL will for example even automatically add obvious indexes if you let it). But what if the query did need to scan all the row…

> The question that follows would be: how do you know what was intended to be less performant versus optimized on-demand? I'm saying that the cloud provider shouldn't try to make assumption either way, and I'm definitely not saying that it should try to manage indexes for you. If you are typically using X ops/s, and begin using 50X ops/s, the default should not be "this customer probably wants to spend 50x their prev…

Honestly, just a limit isn't bad, Just a option to "Stop all operations if bill exceeds 300$" would make this a LOT safer for most folks.

Re: Spending $5k to learn how database indexes work

#178

Earlier quoted context omitted.

> The question that follows would be: how do you know what was intended to be less performant versus optimized on-demand? I'm saying that the cloud provider shouldn't try to make assumption either way, and I'm definitely not saying that it should try to manage indexes for you. If you are typically using X ops/s, and begin using 50X ops/s, the default should not be "this customer probably wants to spend 50x their prev…

Honestly, just a limit isn't bad, Just a option to "Stop all operations if bill exceeds 300$" would make this a LOT safer for most folks.

Or perhaps a “do not allocate more than $1/min” or something similar - which makes cloud servers mimic bare metal hardware - when you overload it slows down but keeps trying.
Post reply on HN