Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The many lies about reducing complexity part 2: Cloud

ea.rna.nl

Post reply on HN
Print view View original

101–110 of 131 posts

Re: The many lies about reducing complexity part 2: Cloud

#101

Post by twistedpair »

arendtio wrote: post #26

I wonder who tells the story that cloud computing has something to do with reducing complexity. In my world, cloud computing is a bout scalability and making things as complex as they need to be to be scalable. This rarely means that complexity is being reduced.

But doesn't cloud computing reduce complexity? I used to write my own SystemV boot scripts, service installation and discovery logic, round robin updating tooling, and much more to handle live app deploy/cut over.

Now I have two dozen lines of YAML and GKE JustWorks™, deploying my services and taking care of everything I used to do with my bespoke scripts. My world is much simpler now.

Re: The many lies about reducing complexity part 2: Cloud

#102

Post by marvinblum »

candiddevmike wrote: post #29

I see a lot of mentions in the comments about just using the basic storage/networking/compute from AWS/AZ/GCP--if that's all you're using, you should really consider other providers. Linode, Digital Ocean, and Vultr will be far more competitive and offer faster machines, cheaper, and with better bandwidth pricing. The point of using AWS/AZ/GCP is to leverage their managed service portfolio and be locked in. If you ar…

This. I switched to Hetzner Cloud and reduced costs from ~$200/month on GCP to about ~20€/month. For comparison: 1cpu 2gb ram cost about $15 on GCP and 2,50€ on Hetzner, 1gb outgoing traffic ~0,12€ -> 20TB per machine included on Hetzner and 1€ per additional TB.

I'm still using S3 and GCS to store files because it's convenient and relatively cheap.

Re: The many lies about reducing complexity part 2: Cloud

#103

Post by bird_monster »

Silhouette wrote: post #92
bird_monster wrote: post #74

Earlier quoted context omitted.

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…

Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. This is true, but it is balanced by the fact that uncertainty can be insanely expensive . And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right (and not find out otherwise…

> And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right

I do not agree. The entire point of using cloud offerings as opposed to rolling your own, is cloud offerings are usually several orders of magnitude easier to configure. Using Event Hub, as an example, means that you're getting a similar experience to Apache Kafka, but without having to scale/configure everything yourself.

Sure, you have to become proficient with Event Hub, but becoming proficient with EH is probably 1/100th of the difficulty as becoming proficient enough in Kafka to support the same scalability/workload.

Re: The many lies about reducing complexity part 2: Cloud

#104

Post by bird_monster »

tpxl wrote: post #84
bird_monster wrote: post #74

Earlier quoted context omitted.

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…

We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it. Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.

You've just wrapped a very specific definition of "Operational support" into a very vague term.

From another post in this thread: If you want to deploy distributed stream processing like Apache Kafka, but do not want to roll it yourself, you can use a tool like AWS Kinesis or Azure Event Hubs.

You still need "Operational Support" to manage EH/Kinesis, but generally it's closer to the type of "Operational Support" that can be provided by a general backend software engineer, as opposed to a DevOps/Infrastructure-specific dev. By using a Cloud (Managed) service, you're removing the need to manage:

* Uptime management * Scaling * Custom-multi-AZ

And probably a lot more. Sure, you've still have to actually handle events appropriately, but you have to do that either way.

Re: The many lies about reducing complexity part 2: Cloud

#105

Post by Silhouette »

bird_monster wrote: post #103
Silhouette wrote: post #92

Earlier quoted context omitted.

Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. This is true, but it is balanced by the fact that uncertainty can be insanely expensive . And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right (and not find out otherwise…

> And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right I do not agree. The entire point of using cloud offerings as opposed to rolling your own, is cloud offerings are usually several orders of magnitude easier to configure. Using Event Hub, as an example, means that you're ge…

All I can say is that this hasn't been my experience. Setting up a single server is much the same whether it's on-prem or at a colo facility or some VM in the cloud, but the amount of added complexity to set up non-trivial networking and security and redundancy/failover and backups and all that stuff in the cloud is far more complicated -- if you don't already know how to do it -- than just setting up a few servers directly. The only exceptions IME tend to be the most basic and essential services, like effectively infinitely scalable storage and managed databases. Not coincidentally, I suspect, these are the kinds of cloud services that almost everyone uses, and often (as this very HN discussion demonstrates) the only ones.

There is still considerable value in the on-demand nature of cloud hardware, instead of waiting for stuff like physically assembling new servers and then shipping them and then installing them in the rack all before you can even start setting up the software, but IME the simplicity emperor has no clothes. Just look at the number of anecdotes about even quite large and well-established businesses that have had systems go down because they didn't fully understand AZs and failover arrangements, or have been landed with some crazy high bill because they didn't fully understand all the different things they were going to be charged for, or have been breached because they left some S3 bucket open.

Re: The many lies about reducing complexity part 2: Cloud

#106

Post by zmmmmm »

jbmsf wrote: post #93
zmmmmm wrote: post #82

Earlier quoted context omitted.

> The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate This sounds like cloud vendor kool aid to me. Nearly every cloud vendor product above the infrastructure layer is a version of something that exists already in the world. When you outsource management of that to your cloud vendor you might lose 50% of the need to operationally manage that product…

I think it's economy of scale, not kool-aid. I can run PostgreSQL myself but there's a ton that goes into running it with redundancy, backup, encryption, etc that takes deep expertise to well. I know from experience that it's easy to get database failover wrong. I could probably cobble something together, but it would be mediocre and wouldn't handle network partitions reliably. On the other hand, RDS is used at a sca…

To be honest, I would probably fit RDS more into the infrastructure layer than the application layer. I think there is value in having things at that level be managed.

Re: The many lies about reducing complexity part 2: Cloud

#107

Post by rubatuga »

nelsonenzo wrote: post #37

This person has never worked in a data center. He thinks he's managing a network because he sets a few vpc ips, that's an itsy tiny fragment of networking, and the cloud has indeed removex a great deal you previously had to manage on prem.

I'm guessing you mean things like STP, firewalls, rate-limiting, routing, DHCP, and link aggregation?

Re: The many lies about reducing complexity part 2: Cloud

#108

Post by bird_monster »

Silhouette wrote: post #105
bird_monster wrote: post #103

Earlier quoted context omitted.

> And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right I do not agree. The entire point of using cloud offerings as opposed to rolling your own, is cloud offerings are usually several orders of magnitude easier to configure. Using Event Hub, as an example, means that you're ge…

All I can say is that this hasn't been my experience. Setting up a single server is much the same whether it's on-prem or at a colo facility or some VM in the cloud, but the amount of added complexity to set up non-trivial networking and security and redundancy/failover and backups and all that stuff in the cloud is far more complicated -- if you don't already know how to do it -- than just setting up a few servers d…

> but the amount of added complexity to set up non-trivial networking and security and redundancy/failover and backups and all that stuff in the cloud is far more complicated

This complexity exists either way, is my point. Whether you're managing your own servers, or using barebones cloud VMs, or using a bunch of cloud fanciness, the complexity you just defined still exists. And if that complexity is a constant, why is it only being used as a negative against cloud services?

> Just look at the number of anecdotes about even quite large and well-established businesses that have had systems go down because they didn't fully understand AZs and failover arrangements, or have been landed with some crazy high bill because they didn't fully understand all the different things they were going to be charged for, or have been breached because they left some S3 bucket open.

If your argument is "It's not better when done badly", definitely, I agree, because what is?

I guess, my overall point is that cloud-based infrastructure shifts your focus. Yes, you have to know how to configure cloud resources, but in 2021, do you think it's easier to find people with AWS experience, or people with custom in-house or colo server management experience?

Re: The many lies about reducing complexity part 2: Cloud

#109

Post by ak217 »

bob1029 wrote: post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

You seem to have made up your mind, but for the benefit of others: yes, we use many AWS products and are having a fantastic time of it. AWS services are more reliable, less buggy and have more stable APIs than any of the alternatives. Specific services that made a difference to us are Fargate, Spot, Batch, SQS/SNS, Lambda, ECR, EFS, ELB/ACM, CloudWatch Logs, IAM, Aurora RDS, SSM/Secrets Manager, and Step Functions - in addition to EC2/VPC/EBS, Route53 and S3 that you already listed. Each of these services does a lot to free us up to do more value added domain-specific work.

Terraform has emerged as a key tool to manage AWS resources - so much so that it really adds a lot to the value prop of AWS itself. I can do stuff with Terraform that was only aspirational until it existed.

Personally, I wouldn't plan on using on-prem except for niche applications that involve heavy data streams from local hardware. In the time that I've spent with companies working on AWS, I've seen a number of other companies waste lots of time and resources on heterogeneous strategies while complaining about their AWS bill - which was high because someone got so fed up with IT dysfunction, they went and used AWS but left behind inefficiently configured resources that were on all the time. Cloud often ends up being an escape hatch for teams that are not adequately served by dogmatic IT departments.

Re: The many lies about reducing complexity part 2: Cloud

#110

Post by dimitrios1 »

ak217 wrote: post #109
bob1029 wrote: post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

You seem to have made up your mind, but for the benefit of others: yes, we use many AWS products and are having a fantastic time of it. AWS services are more reliable, less buggy and have more stable APIs than any of the alternatives. Specific services that made a difference to us are Fargate, Spot, Batch, SQS/SNS, Lambda, ECR, EFS, ELB/ACM, CloudWatch Logs, IAM, Aurora RDS, SSM/Secrets Manager, and Step Functions -…

The key glue to a lot of what you mentioned missing in your list to make a lot of this work for outside consumers: API Gateway, arguably their worst product.
Post reply on HN