Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Saving months of compute time with a single Grafana query

checklyhq.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Saving months of compute time with a single Grafana query

#41

Post by wongarsu »

candiddevmike wrote: post #18
BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.

If you run your own datacenter maybe. But if you pay to rent bare metal servers, orchestrating those isn't any more complex. The biggest downside is that depending on the provider you might have to wait hours instead of minutes when provisioning a new server

Re: Saving months of compute time with a single Grafana query

#42

Post by helsinkiandrew »

mgkimsal wrote: post #34
helsinkiandrew wrote: post #30

Earlier quoted context omitted.

The problem wasn't between the cloud and self hosting - the problem was they had stateful code that didn't scale to thousands of requests for different clients. So they are bringing up new instances every invocation. The same 3s runtime startup cost (and need for more hardware) would happen if they were running their own servers.

Would the actual costs have been less, the same, or more, running on their own hardware? Processes can take longer running on your own hardware, but still have a lower TCO.

Possibly - the advantage of cloud is you don't need to provision for your peak load, it scales with usage from a few requests a day, to millions a second for an hour a day. If your usage is lumpy or growing fast then paying 2-3 times more than the cost of your own hardware can be cheaper.

Re: Saving months of compute time with a single Grafana query

#43

Post by crummy »

sebstefan wrote: post #21

I really don't understand spinning up a whole pod just for a request Wouldn't it be cheaper to just keep a pod up with a service running? If scaleability is an issue just plop a load balancer in front of it and scale them up with load but surely you can't need a whole pod for every single one of those millions of requests right? > Checkly is a synthetic monitoring tool that lets teams monitor their API’s and sites co…

The article said they had to do a bunch of cleanup between requests when it was handled by one service. Which surprised me but these requests must be doing more than just HTTP requests I guess.

Re: Saving months of compute time with a single Grafana query

#44

Post by dxbydt »

many of the tricks we learned in the late 90s - 2000s can no longer be pulled off. We used to download jar files over the net. Running a major prop trading platform meant 1000s of dependencies. You’d have swing and friends for front end tables, sax xml parsers, various numerical libraries, logging modules- all of this shit downloaded in the jar when the customer impatiently waited to trade some 100MM worth of fx. We learned how to cut down on dependencies. Built tools to massively compress class files. Tradeoff 1 jar with lots of little jars that downloaded on demand. Better yet, cache most of these jars so they wouldn’t need to download every single time. It became a fine art at one point - the difference between a rookie and a professional was that the latter could not just write a spiffy java frontend, but actually deploy it in prod so customers wouldn’t even know there was a startup time - it would just start like instantly. then that whole industry just vanished overnight- poof!

now i write ml code and deploy it on a docker in gcp and the same issues all over again. you import pandas gbq and pretty much the entire google bq set of libraries is part of the build. throw in a few stadard ml libs and soon you are looking at upwards of 2 seconds in Cloud Run startup time. You pay premium for autoscaling, for keeping one instance warm at all times, for your monitoring and metrics, on and on. i am yet to see startup times below 500ms. you can slice the cake any which way, you still pay the startup cost penalty. quite sad.

Re: Saving months of compute time with a single Grafana query

#45

Post by c0brac0bra »

crummy wrote: post #43
sebstefan wrote: post #21

I really don't understand spinning up a whole pod just for a request Wouldn't it be cheaper to just keep a pod up with a service running? If scaleability is an issue just plop a load balancer in front of it and scale them up with load but surely you can't need a whole pod for every single one of those millions of requests right? > Checkly is a synthetic monitoring tool that lets teams monitor their API’s and sites co…

The article said they had to do a bunch of cleanup between requests when it was handled by one service. Which surprised me but these requests must be doing more than just HTTP requests I guess.

Yea they do E2E checks with playwright as well, among other things. A bunch of stuff could get cached from those checks I suppose, especially if it's user-written code.

Re: Saving months of compute time with a single Grafana query

#46

Post by sumtechguy »

bravetraveler wrote: post #7
BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

'accept vendor lock in, it'll save you the cost of engineers' Routinely: oops, our API usage slipped and we mistakenly paid more than the staff to avoid this would cost Keep fucking up, tech industry. My job role depends on it (SRE)

Managing these things is a skillset too. You now have X VM's and Y' containers and Z storage things. You still get to manage them. It is easier but is not a zero cost which some people seem to think it is. I have one where it is basically internal and I am at my teams all the time 'clean up your mess'. Tons of PoC's spun up and just left laying around. Things that do millions of calls (to be fixed later). That sort of thing. A cloud abstracts one set of skills. But everything above that line is still on the groups to manage.

Cloud stuff is really alluring at first. Works for awhile then the costs become above what it would cost to run it yourself. Cloud is not a 'set it and forget it' sort of thing. You have to manage it too.

Re: Saving months of compute time with a single Grafana query

#47

Post by wisemang »

belter wrote: post #20
JackMorgan wrote: post #12

Earlier quoted context omitted.

Maybe even shunning the reasonable Price of Education while paying the significant Costs of Ignorance.

AWS has a three day course on the technicals of efficient cost management. Believe it or not...I heard is one of the least requested classes....

Maybe by the time you need it you can’t afford to pay for it…

Re: Saving months of compute time with a single Grafana query

#48

Post by michpoch »

vb-8448 wrote: post #28
hibikir wrote: post #16

Earlier quoted context omitted.

The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necess…

> which often lead to taking months to provision anything, even a very boring VM It's still true? In my experience it used to be, nowadays most of the organizations have an internal serf-provisioning portal.

> organizations have an internal serf-provisioning portal

Which will create things in cloud…

Re: Saving months of compute time with a single Grafana query

#49

Post by michpoch »

trollbridge wrote: post #36
candiddevmike wrote: post #18

Earlier quoted context omitted.

> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.

It really isn’t. I use a combination of bare metal, VMs on those bare metal, and servers hosted at places like Digital Ocean. Orchestration is dead simple and mostly automated using off the shelf, open source tools. If a server goes down, it’s a few minutes to replace it. The cloud based hosting is a fixed cost each month - no usage based surprises. Meanwhile, for clients, spent huge amounts of time fixing broken Kub…

> If a server goes down, it’s a few minutes to replace it.

Like you drive to the server rooms and have a stack of new servers, physically replace the old one with the new one and re-set everything in a few minutes? Or is your “bare metal” an ec2 instance?

Re: Saving months of compute time with a single Grafana query

#50

Post by skrtskrt »

roboben wrote: post #5

Sadly Grafana (cloud) comes at a cost too. Anyone struggles with this horrible active metrics based pricing too? Not only Grafana Cloud but others do it like that too. We moved shitloads to self hosted Thanos. While this comes with its own drawbacks obv, I think it was worth it.

you can self host all the Grafana solutions too if you liked them but didn't like the pricing
Post reply on HN