Live data from Hacker News

AWS costs every programmer should know (2019)

david-codes.hatanian.com

61–70 of 163 posts

Re: AWS costs every programmer should know (2019)

#61
post #4

It's interesting how AWS can keep so high prices on these. But it's just the beginning, the real money comes from when they convince you to run over a dozen vms/containers (all needing storage etc of course). You need to be triply redundant on 3 availability zones, (3x) both with the RDS db cluster and app containers (2x) . And then have separate dev/staging/prod envs (3x). That's 18x. You can then get a pat on the h…

This isn't the only way, though.

Cloudfront can distribute static content and backends running on Lambda@Edge.

DynamoDB supports global distribution (eventually consistent).

If you don't have massive scale, they're quite cheaper than self managing (infra itself and ops people involved).

Yes, these services require adaptation to your old habits of freely controlling your own servers.

As for vendor lock-in, even on EC2 you'll end up with some. Difficult to argue how much, there's no reproducible studies on that.

There are programming practices to minimize the costs of vendor switch as well.

Re: AWS costs every programmer should know (2019)

#62

Earlier quoted context omitted.

Even if you don't want it sometimes you're forced to run multiple AZs (eg: EKS requires 2x). But that 18x figure is nuts. VPCs, AZs, subnets, IAM etc are free. The cost comes from what you deploy into them. So separate environments don't have to be as expensive as production. You can scale them to zero, use smaller computer instances, run self-managed versions of expensive stuff (like DBs) or simply run small single-…

Whilst the VPC itself is free, they get you on the NATGW - which you probably need along with the VPC in most cases.

Had to use a NATGW temporarly on AWS Lambda's to access the database and make remote http calls. But now it all works without the NATGW. Haven't had any other need for one.

Re: AWS costs every programmer should know (2019)

#63

Earlier quoted context omitted.

> And don't even get me started on how much work by this time has gone to build the infra-as-code to manage the rube golderg machine, you'll (seriously) have more lines of CDK code than app logic (that was slower to develop & harder to debug than your actual app code per line). Of all the AWS complaints, CDK is one of yours? I absolutely love CDK and its take on infra-as-code where you construct OO constructs imperat…

CDK code also isn't executed at runtime . Even if it somehow is slower to develop and harder to debug (maybe if you're new to it), it's ideal that you have more of it than actual app code per line. That implies you have substantially less app code than you otherwise would, which (of course) is the entire purpose of these cloud-based abstractions. The amount of app code involved to stitch Kinesis, Lambda, and S3 toget…

> The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it.

The amount of application code to write to a filesystem is... zero.

The amount of application code required to setup a webserver in Java Spring is... zero.

Not sure what Kinesis does, but the amount of application code required to connect to a database is zero (spring), to connect to a rabbitmq/kafka is zero (spring).

And the best part is, it is less complex, mostly open source, and locally runnable compared to the Amazon rube goldberg machine.

Re: AWS costs every programmer should know (2019)

#64
post #51

Earlier quoted context omitted.

This doesn't feel like a meaningful distinction. You can go from an empty cloud account to a fully-provisioned application stack in minutes with the right infra-as-code. Provisioning infrastructure at that level is mutually exclusive with self-hosting. But true, even with self-hosting, you can use infra-as-code at various levels (Ansible, OpenStack, self-hosted Kubernetes). It's just not really comparable to having y…

Who cares when you can accidentally bankrupt your company with the wrong keystroke? Time to spin up is nice for prototypes, it can be very freeing. The mistake that people make is assuming AWS can host things better than they can, since they’re only mere mortals: it leads to a sort of learned helplessness where you assume the cost of a thing is it’s true value and not an egregious markup. My biggest gripe with these…

> Who cares when you can accidentally bankrupt your company with the wrong keystroke?

I think it's fair to say millions of AWS customers manage to avoid bankruptcy quite successfully. If you're incapable of following basic best practices, setting billing alarms, and using IaC, I agree AWS is probably not the best choice for you.

> The mistake that people make is assuming AWS can host things better than they can, since they’re only mere mortals: it leads to a sort of learned helplessness where you assume the cost of a thing is it’s true value and not an egregious markup.

Of course they can--they do it all day every day at Internet scale. I don't want to build S3 everywhere I go. I just want to use it. You could call it a "learned helplessness" in the same way that I don't want to build my own RDBMS, either. I'd rather just use Postgres. This is just the next level of abstraction.

> My biggest gripe with these cloud providers (except maybe GCP) is that they promise less ops work and it can be true in the beginning; but over time the ops work becomes basically the same burden except esoteric to the provider.

It really depends on your team and architecture. If you try to be as cloud-agnostic as possible and abstract away AWS from your devs, then absolutely: you're going to be churning through a lot of ops work that feels like it could be done elsewhere. The more cloud-agnostic you try to be, the less value you'll get from AWS. I've seen numerous companies learning this with EKS (hosted Kubernetes).

That's not the only approach, though. The managed services (like S3, Lambda, DynamoDB, and Kinesis) really add a whole lot of value with substantially less code if your devs are willing to use them. They can even cost less, and especially so if you factor in time spent toward development and building/maintaining alternatives.

> But overall, if you know your problem then time to create an instance isn’t very valuable, believe it or not: the majority of workloads are not excessively elastic, at least the upper bound is not unlimited like many people seem to claim.

A lot of time you don't know your problem, and the minimum instance size for a dev environment in a cloud-agnostic architecture can be significant. Scale to zero is a big help not just in deployment time, but also developer productivity and autonomy.

> Apart from that !cloud != self-hosted. There’s plenty of hardware providers that can get you a dozen machines in under an hour; even discounting virtual host providers like vultr and Tulsa.

I acknowledge they exist, but I do think they're a fraction of the market for good reason. The value adds of cloud providers often outweigh the costs and complexity of the undifferentiated heavy lifting needed for necessary feature parity.

Re: AWS costs every programmer should know (2019)

#65
post #41

AWS is overrated. Most of the engineers and managers choose it because it's a safe choice for them. But in the long term the costs rise and you can't get away from it. I would even argue it's only for those who are afraid of command line.

Cloud should just be a commodity, where you can switch between providers without changing a line of code. Until this is the case, I will keep building my own solutions.

Re: AWS costs every programmer should know (2019)

#66

Earlier quoted context omitted.

Perhaps I am confused or we are talking about different things and consequently talking past each other, but I can literally open a new AWS/GCP/Hetzner/DO account, plug the credentials into my local configuration (in code), and then run a command with NixOps to provision an entire network of machines with custom specifications, and to automatically install all the software I need on those machines. Perhaps you aren't…

It sounds like we're using different definitions of the word "self-hosted." If you have your own on-premises lab or colo rental, you're not using AWS/GCP/Hetzner/DO, and you've got a lot of undifferentiated heavy lifting before your NixOps kick in (including maintenance going forward). If your point is that you can avoid a serverless architecture, still use cloud, and still use infra as code: of course you can. We've…

Hetzner offers colo rental ;)

That said, if you just rent dedicated servers from them, you don’t have to worry about maintenance, but don’t have to pay the ridiculous cloud markups either.

Re: AWS costs every programmer should know (2019)

#67

Earlier quoted context omitted.

If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.

>But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds. Have you actually seen this work on your project in practice ? Like a region go down and another region picks up automatically and it kept working just…

Multi AZ is multiple availability zones. Not multi region. Distribution over multiple regions is obviously harder than within the same region and different zones.

Re: AWS costs every programmer should know (2019)

#68
post #63

Earlier quoted context omitted.

CDK code also isn't executed at runtime . Even if it somehow is slower to develop and harder to debug (maybe if you're new to it), it's ideal that you have more of it than actual app code per line. That implies you have substantially less app code than you otherwise would, which (of course) is the entire purpose of these cloud-based abstractions. The amount of app code involved to stitch Kinesis, Lambda, and S3 toget…

> The amount of app code to solve the same problems without them is orders of magnitude more plentiful and complex, which is why we avoid it. The amount of application code to write to a filesystem is... zero. The amount of application code required to setup a webserver in Java Spring is... zero. Not sure what Kinesis does, but the amount of application code required to connect to a database is zero (spring), to conn…

There will be substantial amounts of code you are writing to provision, set up, maintain, scale, and troubleshoot all of those things. Whether you call it app code or not, it's something that you (and your team) will be on the hook for building and maintaining over time.

Getting your Java Spring web server to scale to thousands of concurrent instances will require a lot of undifferentiated work outside the scope of Java Spring. Same with the care and feeding of RabbitMQ and Kafka. And to suggest that's the approach that's less of a Rube Goldberg machine is... questionable.

Re: AWS costs every programmer should know (2019)

#69
post #12

Earlier quoted context omitted.

If you're not testing az failovers you're probably just wasting money, like untested backups... But it's true that most people don't know this and aren't testing it because they don't know that az failovers don't automatically just work(tm). And the second downside of course would be asymmetry between the envs and divergence in the IaC etc resulting in more complexity and engineering work. (Of course you're probably…

If you’re building your own infrastructure in a data center then sure you absolutely want to test your redundancy. But with AWS it’s a checkbox. It’s transparent to you and your applications. The infrastructure to host in multiple AZs is already in place. The only real issue with MultiAZ is the failover in RDS depending on the database you use could be seconds or 10s of seconds.

Back in colo days, I saw a lot of post-mortems that read “the thing we thought was redundant wasn’t”, leading me to call them “redundant like an appendix [rather than like a kidney]”.

We instituted quarterly “game day testing” where we forcibly turned off one of the redundant items in all of our systems. It took us about 6 such cycles before these tests didn’t turn up outages that were just waiting for us.

Thinking back on those, it’s hard for me to believe that most cloud hosted companies are prepared by checking a box without actually testing.

Re: AWS costs every programmer should know (2019)

#70
post #47
post #28

Earlier quoted context omitted.

> scale to zero whenever possible (lambda) AWS pricing is great because you aren't price-gouged if you don't use it? And your statement isn't really true. If you use AWS Lambdas do implement something like an HTTP endpoint, you still need to pay for stuff like AWS KMS and API Gateway even if no client hits you with a request. - use minimal infra (lambda managing ec2 with route53 health check) Above free tier AWS Lamb…

> Above free tier AWS Lambdas are terribly expensive when compared with EC2 alone. You're charged per RAM*time, you're charged per concurrent execution, and if your Lambda times out you have to pay for the re-execution. They are dirt cheap for many use-cases where you application only runs a few times a day/hour and you need ad-hoc scalable performance. For example processing/creation of csv-reports - which is a comm…

> They are dirt cheap for many use-cases where you application only runs a few times a day/hour and you need ad-hoc scalable performance.

If your app consists of running a quick cron job a couple of times per day, you do not have much of an app to begin with.

If you are already paying for EC2, running those short-lived cron jobs in any one of those instances already adds zero cost.

The main beneficiary of AWS Lambdas is AWS itself. It charges a huge premium for what essentially amounts to spare cycles in its infrastructure, and in the process ties up clients to a proprietary solution.

Post reply on HN