71–80 of 131 posts
Earlier quoted context omitted.
How so?
Less hiccups and downtime. It's faster and with better latency to other third party services. Superior internal control. Ex: In, linode a private IP address gives EVERYONE on the same data center access to your Linode server. Also, last time I used them they didn't have a Firewall.
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…
Maybe not your business, but there are many businesses in which this is exactly what happens. Any managed-service is just combining other people's work into a "product" that gets sold to customers. And that's great! AWS has a staggering amount of products, and lots of business don't even want to have to care about AWS.
> Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it?
Several times. I think cloud products are just tools to get you further along in your business. Most of the tools I use are distributed systems tools, because I don't want to have to own them, and container runtimes/datastores. Every single thing I've ever deployed across AWS/Azure is used as a generic interface that could be replaced relatively easily if necessary, and I've used Terraform to manage my infrastructure creation/deployment process, so that I can swap resources in and out without having to change tech.
If, for some reason, Azure Event Hub stopped providing what we needed it for, we could certainly deploy a customized Kafka implementation and have the rest of our code not really know or care, but from when we set out to build our products, that has always been a "If we need to" problem, and we've never needed to.
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…
I've worked with a number of teams over the last few years who use AWS and I'd say from top to bottom they all build their strategy more or less the same way: 0. Whatever is the minimum needed to get a VPC stood up. 1. EC2 as 90%+ of whatever they're doing 2. S3 for storing lots of stuff and/or crossing VPC boundaries for data ingress/egress (like seriously, S3 seems to be used more as an alternative to SFTP than for…
Earlier quoted context omitted.
I'm not very familiar with AWS or The Cloud, but I'm having trouble understanding what you said about Amazon leaving money on the table by not directing customers toward specific-purpose services as opposed to EC2? Wouldn't (for AWS to make a profit anyway) whatever managed service have to be cheaper than some equivalent service running on an EC2 VM? I get the concerns re: pricing and predictability, but it still see…
For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the enginee…
In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expensive, if using it allows your engineers to focus on building whatever makes _your business successful_, it is usually a much better idea to just use Airflow and keep moving. Clients usually will not care about whether you implemented your own version of an AWS product (unless that's your company's specific business). Clients will care about the features you ship. If you spent a ton of time re-inventing Airflow to save some cost, but then go bankrupt before you ever ship, rolling your own Airflow implementation clearly didn't save you anything.
Earlier quoted context omitted.
I've worked with a number of teams over the last few years who use AWS and I'd say from top to bottom they all build their strategy more or less the same way: 0. Whatever is the minimum needed to get a VPC stood up. 1. EC2 as 90%+ of whatever they're doing 2. S3 for storing lots of stuff and/or crossing VPC boundaries for data ingress/egress (like seriously, S3 seems to be used more as an alternative to SFTP than for…
+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle…
People don't like to admit it, but in many circumstances, having a service that is escalating to 10x or 100x its normal demand go off line is probably the desirable thing.
Earlier quoted context omitted.
+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle…
This seems like you didn't have proper monitoring and alerting set up for your job, not sure how that is a downside of AWS.
"But look at how many monitoring solutions they have in the dashboard! Why, just last re:invent they announced 20 new monitoring features!"
They make a big fuss and show about improving monitoring but it's always crippled in some way that makes it easy to get wrong and time-consuming or expensive to get right.
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and…
Reduced time to market is incredibly valuable. Current client base is well in its millions. Ability to test to few and roll out to many instantly is invaluable. You no longer have to hire competent software developers who understand all patterns and practices to make scalable code and infrastructure. Just need them to work on a particular unit or function.
The thing which scares me is, some of these companies are decades of years old, hundreds. How long has AWS/GCP/Azure abstractions been around for? How quick are we to graveyard some of these platforms. Quite. A lot quicker than you can lift, shift and rewrite your solution to elsewhere.
Earlier quoted context omitted.
Certain services IMHO have to be discounted from this list: - VPC - basic building block for any AWS-based infra that isn't ancient - CloudTrail - only way to get audit logs out of AWS, no matter what you feed them into - CloudWatch - similar with CloudTrail, many things (but not all) will log to CloudWatch, and if you use your own log infra you'll have to pull from it. Also necessary for metrics. - ELB/ELBv2/NLB/ALB…
I don't think you can discount them like that. Maybe they aren't as front of mind as services like S3, EC2, etc, but if you were to try to rebuild your setup in a personal data center, replacing the capabilities of VPC, IAM, CloudTrail, NAT gateways, ELBs, KMS etc would be a huge effort on your part. The fact that they are "basic building blocks" makes them more important, not less. In a discussion about the complexi…
Because they aren't optional, and yes, it takes non trivial amount to replicate them... but funnily enough, several of them have to be replicated elsewhere too.
NAT gateways usually aren't an issue, KMS for many places can be done relatively quickly with Hashicorp Vault.
IAM is a weird case, because unless you're building a cloud for others to use it's not necessarily that important, meanwhile your own authorization framework is necessary even on AWS because you can't just piggy back on IAM (I wish I could).
Earlier quoted context omitted.
For example: Managed Airflow Scheduler on AWS with "large" size costs $0.99/hour, or $8,672/year per instance. That's ~ $17,500 considering Airflow for at least non-prod and prod instances. Building it on your own on same size EC2 instance would cost $3,363/year for the EC2. Times two for two environments, let's say $6,700. $4,000 if you prepay the instance. That looks way cheaper, but then you have to do the enginee…
> That looks way cheaper, but then you have to do the engineering and the operational support yourself. In my experience, this is the piece that engineers rarely realize and that is actually one of the biggest factors in evaluating cloud providers vs. home-rolled. Especially if you're a small company, engineering time (really any employee time) is _insanely valuable_. Valuable such that even if Airflow is cash-expens…
The only caveat is that this goes for founders or engineers who are financially tied with the company success. If the engineer just collects paycheck, they might prioritize fun - and I feel that might be behind a lot of the "reinventing the wheel" efforts you see in the industry.
Or maybe I'm just cynical.
Earlier quoted context omitted.
+1 I bet cloud providers are incentivized not to provide detailed billing/usage stats. I remember having to use a 3rd party service to analyze our S3 usage. Infinite scalability is also a curse - we had a case where pruning history from an S3 bucket was failing for months and we didn’t know until the storage bill became significant enough to notice. I guess in some ways it is better than being woken up in the middle…
This seems like you didn't have proper monitoring and alerting set up for your job, not sure how that is a downside of AWS.
This was the key sentence, I think. This type of problem actually shows up in other domains as well, queueing theory comes immediately to mind. Even the halting problem is only a problem with infinite tape, and becomes easier with (known?) limited resources.
When you have some parameter that is unbounded you need to add extra checks to bound them yourself to some sane value. You are right, in that the parent failed to monitor some infrastructure, but if they were in their own datacenter, once they filled their NAS, I’m positive someone would have noticed, if only because other checks, like diskspace are less likely to be forgotten.
Also, getting a huge surprise bill is a downside of any option, and the risk needs to be factored into the cost. I’m constantly paranoid when working in a cloud environment, even doing something as trivial as a directory listing from the command line on S3 costs money. I had a back and forths with AWS support just to be clear what the order of magnitude of the bill would be for a simple cleanup action since there were 2 documented ways to do what I needed, and one appeared to be easier, yet significantly more expensive.