Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving months of compute time with a single Grafana query

checklyhq.com

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Saving months of compute time with a single Grafana query

#11

Post by bluGill »

bravetraveler wrote: post #7
BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)

Even in Germany you won't get an engineer for only $5k/month. You can get bad engineers in places like India for that, but good engineers cost more than that.

Re: Saving months of compute time with a single Grafana query

#12

Post by JackMorgan »

belter wrote: post #10
bravetraveler wrote: post #7

Earlier quoted context omitted.

'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)

The Tech Industry is well know for shunning the Price of Education, while rejecting the Cost of Ignorance.

Maybe even shunning the reasonable Price of Education while paying the significant Costs of Ignorance.

Re: Saving months of compute time with a single Grafana query

#13

Post by lmz »

Ekrekr wrote: post #6

I really enjoyed this read! One thing that wasn't clear to me, is that if running NPM to install dependencies on pod startup is slow, why not pre build an image with dependencies already installed, and deploy that instead?

Surely they weren't running npm at start. It's just that nodejs allows multiple versions of the same module to coexist and all the different version clients have different version dependencies which could be collapsed to one common version.

Re: Saving months of compute time with a single Grafana query

#14

Post by bravetraveler »

bluGill wrote: post #11
bravetraveler wrote: post #7

Earlier quoted context omitted.

'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)

Even in Germany you won't get an engineer for only $5k/month. You can get bad engineers in places like India for that, but good engineers cost more than that.

German companies for all their precision probably make more than one spending mistake, like we do over here.

There's asymmetry you aren't considering: one engineer can make/solve many problems. Not even all on either side.

To be clear: I'm not taking a dig at this specific case. Larger patterns. It's more nuanced than X of these or Y of those. I'm not arguing in your vacuum/hypothetical

There need-not be an increase in staff in a lot of cases. Just better, or different, staff. If I wanted to earnestly dissect/solve this I wouldn't have opened with snide jokes.

The engineers hired to stave expense tend to go on to create it

The positive spin is: 'get the engineers anyway, self-hosted or not'. They'll help you optimize $solution. My point is things are rarely this positive. Things can be optimized to the point of being non-optimal.

Re: Saving months of compute time with a single Grafana query

#15

Post by tamiral »

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

It’s not a set of up and leave it! You have to continuously monitor and improve! Yes using some cloud service will save XYZ time but doesn’t mean it’s a set it and forget it feature.

I’ll add this is a really good write up ! Love this comment :

“There is no harm in using boring/simple methods if the results are right.”

Re: Saving months of compute time with a single Grafana query

#16

Post by hibikir »

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads.

Until you get quite big, all necessary interactions with the cloud provider are just bills. It's just much easier, even though it is often expensive

Re: Saving months of compute time with a single Grafana query

#17

Post by serverlessmom »

zug_zug wrote: post #8

I'm really surprised that 300ms at startup would result in 25% fewer pods.... What % reduction in the total startup time is that? Is it possible the prior measurement happened during a high traffic period and the post measurement happened in a low traffic period?

It’s a 50% reduction in startup time, and each “run” for a pod is fairly quick.

Re: Saving months of compute time with a single Grafana query

#18

Post by candiddevmike »

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly.

Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.

Re: Saving months of compute time with a single Grafana query

#20

Post by belter »

JackMorgan wrote: post #12
belter wrote: post #10

Earlier quoted context omitted.

The Tech Industry is well know for shunning the Price of Education, while rejecting the Cost of Ignorance.

Maybe even shunning the reasonable Price of Education while paying the significant Costs of Ignorance.

AWS has a three day course on the technicals of efficient cost management. Believe it or not...I heard is one of the least requested classes....
Post reply on HN