Live data from Hacker News

The many lies about reducing complexity part 2: Cloud

ea.rna.nl

91–100 of 131 posts

Re: The many lies about reducing complexity part 2: Cloud

#91
post #75
post #22

Earlier quoted context omitted.

+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle…

> Infinite scalability is also a curse People don't like to admit it, but in many circumstances, having a service that is escalating to 10x or 100x its normal demand go off line is probably the desirable thing.

There seem to be plenty of successful online businesses that share some information about their back end infrastructure, and when you look it turns out they just have a handful of web servers running their app behind a load balancer, enough DB servers for the throughput and redundancy they need and some sort of failover arrangements in case something dies, and some sort of cache and/or CDN arrangements just for efficiency. Even if they're running in the cloud, it's probably just a load of manually allocated EC2 instances, maybe RDS to save the hassle of managing the database manually, and maybe S3.

I wonder how many businesses exist in the entire world that truly need to scale their server counts up or down by a factor of several so quickly that it has to be done automatically and not as a result of, say, daily human reviews of what's happening. I feel like even large businesses that need to run thousands of servers at all are probably relatively rare, as indeed large businesses themselves are. The number that might realistically need to scale from say 1,000 web servers to 1,500 within a matter of hours must be even rarer. But I have nothing even resembling a useful data source to tell whether this intuition is correct.

Re: The many lies about reducing complexity part 2: Cloud

#92
post #69

Earlier quoted context omitted.

For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the enginee…

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…

Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_.

This is true, but it is balanced by the fact that uncertainty can be insanely expensive. And diving into complicated cloud infrastructure with a small business, if you're not already an expert on it, is a very uncertain endeavour in terms of whether you'll get everything set up right (and not find out otherwise at 3am when it turns out your redundancy wasn't, for example) and what everything will cost. By the time you have either become an expert yourself or hired someone who already is, your costs have already increased significantly too, for exactly the reason you've just stated yourself.

Re: The many lies about reducing complexity part 2: Cloud

#93
post #82
post #25

Earlier quoted context omitted.

I can't agree, at least not in general. The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate and because of the functionality you have to exclude from your systems. I am sure there are scales where you either have such a large engineering budget that you can ignore these costs or where decreasing your cloud spend is the only way to scale your busines…

> The optionality of being cloud agnostic comes with a huge cost, both because of all the pieces you have to build+operate This sounds like cloud vendor kool aid to me. Nearly every cloud vendor product above the infrastructure layer is a version of something that exists already in the world. When you outsource management of that to your cloud vendor you might lose 50% of the need to operationally manage that product…

I think it's economy of scale, not kool-aid.

I can run PostgreSQL myself but there's a ton that goes into running it with redundancy, backup, encryption, etc that takes deep expertise to well. I know from experience that it's easy to get database failover wrong. I could probably cobble something together, but it would be mediocre and wouldn't handle network partitions reliably. On the other hand, RDS is used at a scale well beyond what I could afford to build out and has benefitted from much more usage. Problems that are low probability for me are regular events for RDS and have that experience built in.

Ultimately, you are paying a cloud vendor for service value and operational experience. Some services aren't as good as others in both regards, but for the ones that are good, the overhead of doing it yourself is an exponent, not a fraction.

Re: The many lies about reducing complexity part 2: Cloud

#94
post #84

Earlier quoted context omitted.

> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…

We used to have on-prem redis and a devops engineer to manage it, then we moved to redis in the cloud and had a devops engineer to manage it. Saying that in the cloud you don't need engineers to manage "operational support" is the biggest lie the cloud managed to sell.

If you just run redis on virtual server in cloud, you're not replacing the devops engineer who manages redis. You're replacing:

- the network engineers who manage switches, firewalls, routers, nats and connectivity

- the people who manage on-prem hardware, install new servers, replace failing servers, and switch broken disks for working ones and install base os using some ilom

- the whole process for ordering that hardware and getting it delivered on time - including knowing when it's going to be necessary

- if your on-prem had virtual servers, the people who manage and operate vmware

- if your on-prem had SAN, the people who manage and operate that SAN (and buy new disks and take care of capacity planning)

Some of those things you still have to do - for example configure firewall, or say "I want to take a snapshot of this virtual disk drive" but instead of doing a difficult technical thing, you can do it in web ui - but you still need to do what to do.

And of course, if you never had SAN and virtual servers and two data center with your team managing the interconnect between private networks, there's a lot of stuff that the Cloud could give you that you probably don't need.

Now if you move to managed redis, you're also replacing the person who installs and patches the linux the redis runs on, and the one who installs, backups and configures the redis. And you get the redis on button click, so if you suddenly need three more, you're also replacing the person who automates building redises.

You are right, that it is not that the operational support is just gone. Some of it is gone. Some of it is replace by doing cloud stuff (like thinking of buying reserved instances instead of actual physical servers). Some of it is just more efficient.

Now if any of this doesn't fit you're use case because you have too small or too large scale, then the Cloud is of course a bad idea.

Re: The many lies about reducing complexity part 2: Cloud

#95
For simple document or file type storage for an application I think solutions like S3 or azure blob storage are really great. In my real world experience we replaced a highly finicky way too heavyweight COTS CMS and all the associated database and backend SAN complexity with S3 and it ended up being way cheaper, easier and more reliable. To me object storage is one place where the cloud hype does deliver. Have had mixed results with the other services, even with things like RDS you have to do a lot of tuning and backup / DR work yourself.

Re: The many lies about reducing complexity part 2: Cloud

#96
post #85
post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

Did you look into multi-cloud solutions like Pulumi or Terraform to abstract your cloud vendor?

Don't fall for that trap.

I have never used Pulumi, I've used Terraform a bit. I like Terraform so this isn't a dig at the tools.

Abstracting your cloud provider is similar to abstracting your database. At a high-level they appear to be the same, they do similar things however the are very different when you get to the fine detail.

Pulmi / Terraform are useful in the provide a common language / API between cloud providers however you will never just switch cloud provider with changing a few lines of code.

Re: The many lies about reducing complexity part 2: Cloud

#97
post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

It depends on your company. For new startups, I'd stick with delivering your product over building your own cloud services on top of EC2/S3.

At my last startup, the engineering head shared the Not-Invented-Here view. We couldn't use any handy services. We had to run our own Cassandra and everything else services. It was a huge time sink for a small team that didn't deliver differentiating value.

At my current startup, we're almost 100% serverless on SaaS providers. We operate a thousand or more nodes with no ops team (a couple engineers know ops, when needed). We leave the complexity of scaling an maintenance to our cloud vendor's autoscaling services. Sure, we could reduce our opex in places by trying to roll our own, but the ability to have the vast majority of engineers delivering customer product, rather than reinventing the wheel is more valuable to us than fear of vendor lock-in. If our cloud vendor wanted to jack rates, we'd take the necessary action, which we've done before when we banged a service in a week that a vendor was going to hold us over a barrel for, then dropped the vendor.

Re: The many lies about reducing complexity part 2: Cloud

#98
post #4

Earlier quoted context omitted.

There is an element of _how_ as well. You could create simple monoliths or overengineered microservices. Or, complex monoliths with heavy coupling vs cleanly designed microservices with clear separations of concern.

Are microservices meant to separate data too? As in, each service has its own database. Wouldn't that lead to non-normalisation of the data or a lot of expensive network lookups to get what I want/need? What is the point of micro services anyway :-)?

There's a progression ever developer that grew up on SQL Server/relational data needs to go through...

1. The models (plural intended) of a business are not necessarily a relational. There could be an upstream/downstream relationship. There could be an event-based relationship. There may be no relations at all (documents are handy in these scenarios).

Stop assuming you start with an entity relation diagram. That's an immediate limiting factor when listening to the business describe their processes.

2. There is no such thing as immediate updates to any database. There is _always_ latency. Build software understanding this.

3. Operational data and Analytical data are TWO DIFFERENT THINGS. (sorry for the shouting)

Operationally, I only need concern myself with the immediate needs of the user or process. If I'm doing "something" to the customer domain, I don't need to know or do anything else. If I'm doing something to the order domain, I may need to notify some other domains of what I'm doing or have done, but that's secondary and not _immediately important_. Inventory systems should have built-in mechanisms for levels and never need to know the exact up-to-date figures.

My operational domains can notify whole other systems on changes in data. So your analytical system can subscribe to these changes and normalize that data all it wants. I can even build user interfaces that display both operational and analytical data.

Micro-services are brilliant at operational domain boundary adherence. Events are brilliant at notifying external boundaries of change.

The caveat I point out to my clients is that thinking in this way is very different than we're used to and often comfortable with. It takes time to identify the best boundaries and events for the models of a business. But if you put in that time, the result will be software that your business personnel can actually understand.

Re: The many lies about reducing complexity part 2: Cloud

#99
post #6

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…

> The one time somebody tries to use some managed service that goes overbudget by 3000%, and the after action figures out that it would have been within the budget by using in EC2, they just do that instead

This impacts casual dabblers too. More than once I've seen HN comments on how someone is wary to experiment with cloud computing because a single screw-up can lead to an essentially unlimited bill. Judging by HackerNews anecdotes of when this does happen (unexpected overruns of thousands of dollars), there's a reasonable chance Amazon will refund you out of good will, but that's not enough to lay the fears to rest.

Linode let you pre-pay, for instance, but (to my knowledge) this isn't an option offered by Amazon, Microsoft, or Google.

Re: The many lies about reducing complexity part 2: Cloud

#100

Subjectively, it increasingly feels that while the complexity has been increasing, the notion of longevity of the underlying products and services has been degrading. While updates to software were expected, general outlook would be that they would not be breaking the core features. The emphasis on backwards compatibility was in a way an assurance to businesses that building their operations on vendor's products is n…

> the notion of longevity of the underlying products and services has been degrading.

Is eternal longevity even a relevant concept in computing ops any more?

I think businesses must grok that all services are living things that will require eventual maintenance. If you roll your own service, eventually the OS, even an LTS, will need upgrading when the prior release goes EOL. If you're higher up on a FaaS, the language/runtime will still age out, like Python 2, Node.js 6, or Java 5. These will inevitably come with changes. You application will need to adapt.

In extreme cases you can pay for ancient language support and customer IBM mainframe hardware, but the reality is the software's hosting will always need to be updated as time marches on.

Post reply on HN