Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spending $5k to learn how database indexes work

briananglin.me

Post reply on HN
Print view View original

141–150 of 209 posts

Re: Spending $5k to learn how database indexes work

#141

Post by ashleyn »

nrmitchi wrote: post #74

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

There's no free lunch. Cloud services trade performance woes for budget surprises. This may be preferable in some cases but the tradeoff should be recognised.

Re: Spending $5k to learn how database indexes work

#142

Post by Spooky23 »

Nextgrid wrote: post #88
Spooky23 wrote: post #26

Earlier quoted context omitted.

25 years ago i built and ran a biggish database system that supported a reservation system. Even given the limitations of the time (RDBMS cost, 9GB disks, Sun kit, etc), our cost of good sold for that type of workload was exponentially less. (At scale) Today, I could probably run that company off my MacBook Pro and have room to spare. That said, the rationale for choosing this technology is cute: “After seeing a ton…

I have seen so much over-engineered startup shit costing thousands in AWS fees that could run faster on my laptop.

Lol funny that you say that.

I’m not a developer by trade (mostly an email/excel/PowerPoint jockey these days), but am a local sme on a few things, one of which was absolutely critical for a very key project.

I was asked to mock-up a prototype of a core process that produced correct outputs. Dusted off my old toolbox and mocked it up in a combo of python and bash. Probably a total of 900-1000 lines of “code”. The mock-up, running on some little vm, ended up outperforming the production solution for quite some time! :)

Re: Spending $5k to learn how database indexes work

#143

Post by rustc »

cperciva wrote: post #105
XCSme wrote: post #45

Earlier quoted context omitted.

Still crazy, charging $0.15 for probably less than a few ms of computing power.

If those million items need to be read from different servers -- as they might well, in a distributed database -- it's definitely not just a few ms of computing power. For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb.

> For reference, reading a million items of up to 1 kB each costs $0.125 with on-demand dynamodb.

Is that the same counting method as PlanetScale's "row read"? That is, `select title from posts order by title limit 10` on a table with 10 million rows and no index on `title` would cost $1.25 per query?

Re: Spending $5k to learn how database indexes work

#144

Post by [deleted] »

Grimm1 wrote: post #140
ampersandy wrote: post #131

Earlier quoted context omitted.

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. Because it's a well written, humble account of learning from a mistake then using it as an opportunity to teach others to help them avoid the same mistake. If anyone leads a team, I hope they might learn from this approach, rather than just bashing on people and implying they don't deserve any attention because they…

Personally I find it trite but whatever floats your boat.

[deleted]

Re: Spending $5k to learn how database indexes work

#145

Post by nrmitchi »

lmilcin wrote: post #138
nrmitchi wrote: post #74

As the author touches on, the main problem here isn't learning about indexes. It's about "infinity scaling" working too well for people who do not understand the consequences. In no sane version of the world should "not adding a db index" lead to getting a 50x bill at the end of the month without knowing. I am a strong believer that services that are based on "scale infinitly" really need hard budget controls, and sl…

Why not write a simple service that tracks various stats (like number of users, requests, etc.) as well as billed costs over time? You could then get various interesting stats in real time as well as some pretty useful alerting.

Even with "infinite scale", you should still be monitoring, and be doing some form of budget monitoring.

The difference is that application performance metrics are generally available in near-real time, whereas billing metrics are 1) very platform specific, and 2) generally not even close to real time.

It's hard to react quickly when your platform has effectively transformed near-real-time performance alerts to delayed/rolled-up billing alerts (which would also be much more difficult to use to pinpoint where the underlying issue is)

Re: Spending $5k to learn how database indexes work

#146

Post by bsder »

throwawayapples wrote: post #101
amenod wrote: post #90

Earlier quoted context omitted.

...and Hetzner just started offering their services in the US a few days ago. (EDIT: not affiliated) If you do something stupid with your code at least you won't go bankrupt, only your service will be slower.

I was just comparing the pricing to OVH's cloud VPS's ( https://us.ovhcloud.com/ ) and, accounting for the currency conversion and OVH's huge amount of free unmetered bandwidth (vs 20TB for H), it actually looks like OVH is even cheaper. How can DO and Vultr even compete? Probably on the basis of their nicer dashboards and easier sign-up flow (especially OVH's)

I don't know about Hetzner, but Vultr has had quite nice customer support.

When I wanted to do some non-standard things, I simply talked to Vultr to demonstrate that I had a clue, had a reason for asking that I could articulate, and they approved it right then.

Re: Spending $5k to learn how database indexes work

#147

Post by xupybd »

Grimm1 wrote: post #69
xupybd wrote: post #50

Earlier quoted context omitted.

The best technical people aren't always the best to start a business. The goal is to make money not have perfect code.

Cool, and if this was a case of bike shedding over something that hadn't just cost that early stage startup 5 grand I'd agree with you. However regurgitating a platitude that everyone, including myself, learned when we tried getting our first business off the ground doesn't add much value here. Had this been a Database with 10million rows it would have cost them 50k, and this is incredibly basic programming knowledge…

I've been involved in some terrible software behind reasonably successful businesses. I complained like the best of them having to clean up the horrible mess. But it worked they used their limited competence and limited funds then built something profitable.

Re: Spending $5k to learn how database indexes work

#148

Post by Aeolun »

williamdclt wrote: post #132
Nextgrid wrote: post #18

Earlier quoted context omitted.

> I'm not entirely sure why buzz around "developer learns basic knowledge" has this on the front page. The problem is that in the old days, not knowing about indexes left you with an underperforming system or downtime. But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal.

> But in The Cloud™ it leaves you with an unreasonably huge bill and that somehow as an industry we're accepting this as normal. No. Nobody finds that "normal", that's just untrue. It's even the _whole_ subject of this blogpost: the bill was not normal. I don't disagree that some people are overrelying on cloud services, but that didn't become normality, it's still a beginner's mistake

> it's still a beginner's mistake

Absolutely, but previously that wouldn’t cost you €5000 extra.

Re: Spending $5k to learn how database indexes work

#149

Post by Aeolun »

williamdclt wrote: post #136
pjscott wrote: post #84

Earlier quoted context omitted.

One of the best database habits I've ever developed is to run EXPLAIN on every query that I expect to run repeatedly, then sanity-check the output. It's very little effort, and has prevented so much hassle.

If we weren't using underwhelming ORM DSLs, I'd love to use/write a github bot that automatically runs EXPLAIN ANALYZE on queries updated in a PR and post the query plan!

You can do that still. Just a bit more work to evaluate the ORM layer first.

Re: Spending $5k to learn how database indexes work

#150

Post by utopcell »

AnotherGoodName wrote: post #15

I've seen things you people wouldn't believe. Millions burnt on consultants and licensing Oracle. I watched C series startups throwing it all away in a move to NoSQL. All those Amazon RDS fees will be lost in time.

"All those Amazon RDS fees will be lost in time" ..like tears in rain?
Post reply on HN