The many lies about reducing complexity part 2: Cloud • Hacker News Board

Re: The many lies about reducing complexity part 2: Cloud

#81

Post by g9yuayon » Sun, Jan 10, 2021, 9:43 PM UTC

bob1029 wrote: post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

So your company cautiously chooses which services in AWS to use, and sticks to infrastructure offerings for now. Netflix called it "paved path", and it worked really well too for Netflix. Over the years, though, the "paved path" expanded and extended to more services. It's worth noting that EC2 alone is a huge productivity booster, bar none. Nothing beats setting up a cluster of machines, with a few clicks, that auto scales per dynamic scaling policies. In contrast, Uber couldn't do this for at least 5 years, and their docker-based cluster system is a crippled for not supporting the damn persistent volumes. God knows how much productivity was lost because of the bogus reasons Uber had for not going to cloud.

Re: The many lies about reducing complexity part 2: Cloud

#82

Post by zmmmmm » Sun, Jan 10, 2021, 9:46 PM UTC

jbmsf wrote: post #25

maria_weber23 wrote: post #19

Earlier quoted context omitted.

I second that. It's not only that you make yourself completely intertwined with a Cloud by using more than fundamental services. The costs of lambda or even DDB are IMMENSE. These only pay off for services that have a high return per request. I.e. if you get a lot of value out of lambda calls, sure, use them. But for anything high-frequency that earns you little to nothing on its own, forget about it. Generally all y…

I can't agree, at least not in general. The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate and because of the functionality you have to exclude from your systems. I am sure there are scales where you either have such a large engineering budget that you can ignore these costs or where decreasing your cloud spend is the only way to scale your busines…

> The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate

This sounds like cloud vendor kool aid to me. Nearly every cloud vendor product above the infrastructure layer is a version of something that exists already in the world. When you outsource management of that to your cloud vendor you might lose 50% of the need to operationally manage that product but about 50% of it is irreducible. You still need internal competence in understanding that infrastructure and one way or another you're going to develop it over time. But if its your cloud vendor's proprietary stack then you are investing all your internal learning into non-transferrable skills instead of ones that can be generalised.

Re: The many lies about reducing complexity part 2: Cloud

#83

Post by 8note » Sun, Jan 10, 2021, 9:46 PM UTC

ratww wrote: post #39

skohan wrote: post #30

I have to say, at my current company we are using Serverless, and it really does feel like it reduces complexity. No runtime/framework to set up, no uptime monitoring or management required on the application layer, and scaling is essentially solved for us. I mean you do pay for what you get, but it does feel like one of those technologies which really lowers the barrier to entry in terms of being able to release a p…

I used to be very excited about serverless, and I still have high hopes for it. But for me it ended up replacing the complexity of runtime and frameworks with the complexity of configuring auxiliary srevices like Gateway API, Amazon VPC, etc. We needed to move the complexity to some tool that configured the services around Lambda, like Terraform or Cloud Formation, or at best to a framework like Claudia or Serverless…

I was expecting writing serverless to be a mess of writing configuration, but I've really enjoyed writing CDK for cloudformation. It's super unclear how you're supposed to write good cdk code, but I feel like I'm a lot clearer on what infrastructure I'm actually using than before, where I was relying on stuff set up by someone else ages ago with minimal to no documentation

Re: The many lies about reducing complexity part 2: Cloud

#84

Post by tpxl » Sun, Jan 10, 2021, 10:06 PM UTC

bird_monster wrote: post #74

glogla wrote: post #69

Earlier quoted context omitted.

For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the enginee…

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…

We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it.

Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.

Re: The many lies about reducing complexity part 2: Cloud

#85

Post by mycall » Sun, Jan 10, 2021, 10:14 PM UTC

bob1029 wrote: post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

Did you look into multi-cloud solutions like Pulumi or Terraform to abstract your cloud vendor?

Re: The many lies about reducing complexity part 2: Cloud

#86

Post by dijit » Sun, Jan 10, 2021, 10:31 PM UTC

dilyevsky wrote: post #71

ithrow wrote: post #48

Earlier quoted context omitted.

Less hiccups and downtime. It's faster and with better latency to other third party services. Superior internal control. Ex: In, linode a private IP address gives EVERYONE on the same data center access to your Linode server. Also, last time I used them they didn't have a Firewall.

Right linode is basically old school dedicated servers afaik but DO should be in different class

AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests).

Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance.

For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between those nodes often and sometimes congestion which drops users. (IRC is highly stateful TCP sessions).

I've only known two truly good networking suppliers, GCP (and their magic BGPPoPDark Fibre networks) and.. Tilaa.. (which is only hosted in Netherlands.. which is why I can't use them for my global network)

Re: The many lies about reducing complexity part 2: Cloud

#87

Post by dilyevsky » Sun, Jan 10, 2021, 10:35 PM UTC

dijit wrote: post #86

dilyevsky wrote: post #71

Earlier quoted context omitted.

Right linode is basically old school dedicated servers afaik but DO should be in different class

AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests). Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance. For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between…

Awesome thanks for info. For gcp i notice occasional unavailability on the order of 10s of mins every quarter or so. That’s VM networking. Their loadbalancers are a different story as they are a complete crap

Re: The many lies about reducing complexity part 2: Cloud

#88

Post by dijit » Sun, Jan 10, 2021, 10:37 PM UTC

dilyevsky wrote: post #87

dijit wrote: post #86

Earlier quoted context omitted.

AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests). Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance. For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between…

Awesome thanks for info. For gcp i notice occasional unavailability on the order of 10s of mins every quarter or so. That’s VM networking. Their loadbalancers are a different story as they are a complete crap

This is very much not my experience, do you have any more information?

Any particular regions? Are you certain it's not a local ISP?

(I used to run an always online video game and we had a LOOOOOT of connection issues from "Spectrum internet" on all of our servers including GCP ones.)

Re: The many lies about reducing complexity part 2: Cloud

#89

Post by gregmac » Sun, Jan 10, 2021, 10:42 PM UTC

mumblemumble wrote: post #57

hyperdimension wrote: post #27

Earlier quoted context omitted.

I'm not very familiar with AWS or The Cloud, but I'm having trouble understanding what you said about Amazon leaving money on the table by not directing customers toward specific-purpose services as opposed to EC2? Wouldn't (for AWS to make a profit anyway) whatever managed service have to be cheaper than some equivalent service running on an EC2 VM? I get the concerns re: pricing and predictability, but it still see…

It's not just about a straight cost comparison. It's about how organizational decision-making works. The people shopping for products are not spending their own money, but they are spending their own time and energy. The people approving budgets are not considering all possible alternatives, they are only considering the ones that have been presented to them by the people doing the shopping. If the shoppers decide th…

This is a really good summary of the situation, and I'd add a bit about risk:

It's relativity easy to estimate EC2 costs for running some random service, because it's literally just a per-hour fee times number of instances. If you're wrong, the bigger instance size or more instances isn't that much more expensive.

For almost every other service, you have to estimate some other much more detailed metric: number of http requests, bytes per message, etc. When you haven't yet written the software, those details can be very fuzzy, making the whole estimation process extremely risky - it could be cheaper than EC2, it could be 10x more, and we won't really know until we've spent at least a coulple months writing code. And let's hope we don't pivot or have our customers do anything in a way we're not expecting..

Re: The many lies about reducing complexity part 2: Cloud

#90

Post by dilyevsky » Sun, Jan 10, 2021, 10:45 PM UTC

dijit wrote: post #86

dilyevsky wrote: post #71

Earlier quoted context omitted.

Right linode is basically old school dedicated servers afaik but DO should be in different class

AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests). Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance. For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between…

Answering here bc bottom post is locked for some reason - east1 occasionally disconnects from other regions. That is definitely within google backbone. Central-1 seems worse tho. If it’s less than an hour they dont bother with the status page.

For loadbalancer its very much by design as they randomly send you rst when google rolls them for upgrade and in some other cases (I’m working on a blog post on this). Google support recommendation is to retry (foreals)