Saving months of compute time with a single Grafana query • Hacker News Board

Re: Saving months of compute time with a single Grafana query

#31

Post by silverquiet » Fri, Jul 12, 2024, 12:58 PM UTC

hibikir wrote: post #16

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necess…

Cloud was absolutely sold as saving money once upon a time, but like any other industry, they wanted to push the margins higher by getting away from providing commodity services (cloud primitives in their case), and thus the marketing started saying it was never about saving money or how you're not supposed to use the primitives; you need to use the higher-level, managed services (that also happen to have tons of vendor lock-in).

I've spent the last decade or so wondering if the emperor was wearing clothes and not really getting what everyone else has been talking about. Which isn't to say that cloud is useless, but it's not the universal panacea that it was often sold as, and it seems that others are waking up to that.

Re: Saving months of compute time with a single Grafana query

#32

Post by dartos » Fri, Jul 12, 2024, 12:59 PM UTC

vb-8448 wrote: post #28

hibikir wrote: post #16

Earlier quoted context omitted.

The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necess…

> which often lead to taking months to provision anything, even a very boring VM It's still true? In my experience it used to be, nowadays most of the organizations have an internal serf-provisioning portal.

Who do you think builds and maintains those portals?

Internal data center organizations that need to be argued with to acquire quota. They usually don’t care for internal or dev stuff, but as soon as something needs prod levels of quota or new licenses, prepare to wait.

Re: Saving months of compute time with a single Grafana query

#33

Post by latchkey » Fri, Jul 12, 2024, 1:01 PM UTC

serverlessmom wrote: post #27

rjmunro wrote: post #26

Earlier quoted context omitted.

$5k/month was 25% of his pods, so the total was ≈$20k/month. It's entirely possible that self hosting would cost much more than that, particularly as they wouldn't be able to save costs by scaling down.

Yeah. My dad had told me more than one horror story of early tech startups buying truckloads of hardware to scale way beyond demand growth. And I remember when getting on Slashdot meant your service would inevitably go down. Of course, given stable demand and known requirements, bare metal can be a great option. But it’s not strictly better than public cloud hosting. I think it’s just been long enough that people hav…

> I think it’s just been long enough that people have forgotten the limitations of bare metal engineering.

Not just engineering, but deploying it too. I'm building a whole business around deploying it based at least partially on the fact that it has been forgotten. I've just been doing it long enough that I remember how to do it and it isn't getting any easier as the need for compute grows into even more complex and powerful hardware.

Re: Saving months of compute time with a single Grafana query

#34

Post by mgkimsal » Fri, Jul 12, 2024, 1:01 PM UTC

helsinkiandrew wrote: post #30

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

The problem wasn't between the cloud and self hosting - the problem was they had stateful code that didn't scale to thousands of requests for different clients. So they are bringing up new instances every invocation. The same 3s runtime startup cost (and need for more hardware) would happen if they were running their own servers.

Would the actual costs have been less, the same, or more, running on their own hardware? Processes can take longer running on your own hardware, but still have a lower TCO.

Re: Saving months of compute time with a single Grafana query

#35

Post by rbranson » Fri, Jul 12, 2024, 1:01 PM UTC

latch wrote: post #24

candiddevmike wrote: post #18

Earlier quoted context omitted.

> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.

If your scale is crazy, or your product doesn't allow you to use battle-tested pieces, then orchestration is complex in both cases. In most cases, managing software on bare metal is more complex in exactly one case: when engineers only know cloud abstractions.

Yeah calling the EC2 API is definitely more complex than leasing datacenter space, purchasing racks of hardware, deploying a fault tolerant and secure network, capturing and managing offsite backups, dealing with hardware component failures, etc.

Re: Saving months of compute time with a single Grafana query

#36

Post by trollbridge » Fri, Jul 12, 2024, 1:03 PM UTC

candiddevmike wrote: post #18

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

> Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Bare metal and datacenter orchestration is leaps and bounds more complex. You're paying for the abstraction.

It really isn’t. I use a combination of bare metal, VMs on those bare metal, and servers hosted at places like Digital Ocean.

Orchestration is dead simple and mostly automated using off the shelf, open source tools. If a server goes down, it’s a few minutes to replace it. The cloud based hosting is a fixed cost each month - no usage based surprises.

Meanwhile, for clients, spent huge amounts of time fixing broken Kubernetes setups and hit serious design constraints because of usage based pricing on their PaaS infrastructure like being unable to do complex queries from a database.

I wouldn’t think twice about the same query on our in house hosted DBs on $400 servers.

Re: Saving months of compute time with a single Grafana query

#37

Post by [deleted] » Fri, Jul 12, 2024, 1:04 PM UTC

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

[deleted]

Re: Saving months of compute time with a single Grafana query

#38

Post by trollbridge » Fri, Jul 12, 2024, 1:04 PM UTC

rbranson wrote: post #35

latch wrote: post #24

Earlier quoted context omitted.

If your scale is crazy, or your product doesn't allow you to use battle-tested pieces, then orchestration is complex in both cases. In most cases, managing software on bare metal is more complex in exactly one case: when engineers only know cloud abstractions.

Yeah calling the EC2 API is definitely more complex than leasing datacenter space, purchasing racks of hardware, deploying a fault tolerant and secure network, capturing and managing offsite backups, dealing with hardware component failures, etc.

If you’re at that scale, there are plenty of other platforms that don’t have vendor lock in and overly complex, proprietary APIs by design.

Re: Saving months of compute time with a single Grafana query

#39

Post by [deleted] » Fri, Jul 12, 2024, 1:04 PM UTC

[deleted]

Re: Saving months of compute time with a single Grafana query

#40

Post by jermaustin1 » Fri, Jul 12, 2024, 1:07 PM UTC

hibikir wrote: post #16

BobbyTables2 wrote: post #4

I do not understand how cloud proponents talk about the he costs of self hosting but then get into situations like this. Spending serious engineering time to wrangle with the complexities of cloud orchestration is not something that should be taken lightly. Cloud services should be required to have a black-box Surgeon’s General warning.

The best advantage of cloud was never price: It was not having to argue with your data center organization, which often lead to taking months to provision anything, even a very boring VM. If those companies were good at managing data centers, and could hire people actually interested in helping the company run, they'd have had little need for the cloud in predictable compute loads. Until you get quite big, all necess…

The other advantage was for moving off-prem. Cap-Ex vs Op-Ex. That was the reason one of my past employers switched.

Another past employer switched because we hit our scale up limit, and needed to start scaling out. A small refactor allowed us to scale out, and we moved to azure's managed database, queue, and blob storage. The web frontend could scale based on connections, and the queue and blob storage was slower than our current approach, but it was better once we added the autoscaling. Since the slower speed was PER connection. Minimum scale was 5 instances, so that there was no bottleneck when scaling.

There are many reasons to go "cloud" but for most small businesses (or at least small departments of large organizations), cloud-first doesn't seem like a great option unless you have 10s of thousands in credits each month. Just build your software, scale up first on-prem or at a datacenter - it is LOADs cheaper and predictable.