I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…
Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…
We reduced our server costs by moving away from AWS
161–170 of 355 posts
Re: We reduced our server costs by moving away from AWS
#162I've said this a hundred times and it seems not loud enough. AWS is not cheap because of your server costs. AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours. "The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (a…
Elasticity is a gamble. You're betting that you can save more money investing in an elastic/on demand stack than what you'd pay for static resources. Judging by how much the cloud providers push this, the unknown cost to create/maintain auto scaling infra/apps, and how intricate the pricing gets with elastic resources/spots/faas, I still think the cloud providers are coming out ahead vs folks using static compute.
For a mature business, that tradeoff probably isn't worth it, but for a startup, the opportunity cost is too high. Spending a lot on cost optimization doesn't make sense; you are better off spending on growing income.
Re: We reduced our server costs by moving away from AWS
#163I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…
Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electricia…
Co-locating has no capital investment other than hardware, and is pretty cheap.
A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year.
Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital.
All this to say is that I don’t think capital is as big a factor as you might think.
Re: We reduced our server costs by moving away from AWS
#164This is the Trillion Dollar Paradox described by Martin Casado. You’re crazy if you don’t start your business in the cloud, you’re crazy if you stay there. My new startup is focused on helping application owners repatriate their workloads into their own infrastructure. Our goal is to solve the network complexity challenges with a fully open network stack (open source software with all of the hardware options you woul…
I've been getting by on like 80$/month hetzner servers (Like ancient 10 year old pizza box machines) for my businesses for a long while. Never did the cloud thing, I can run like three or four websites on one of those bad boys. I guess stuff like AWS makes sense if your computational workload happens in bursts or whatever? But 80$/month overhead costs are like nothing, I spend more than that on a good night out level.
Re: We reduced our server costs by moving away from AWS
#165They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce yo…
Yeah, you need to do calculation of what you are doing right now vs AWS. If you already have a mature team that is doing well optimising the environment, you have stable demand and no need to develop things rapidly then it is very likely that going cloud is poor choice. As to "cloud agnostic", don't believe this bullshit. In my opinion most projects fare much better by just letting go of this "cloud agnostic" and jus…
I feel like a variety of other circumstances can come to pass, which would negatively affect business continuity, here's what a lazy search turned up:
"My Google Cloud was suspended too" https://news.ycombinator.com/item?id=32571055
"Google suspended our domain out of the blue" https://news.ycombinator.com/item?id=32798368
"Tell HN: Google Cloud suspended our production projects at 1am on Saturday" https://news.ycombinator.com/item?id=32547912
"AWS account was permanently closed because it was suspended for 90 days" https://news.ycombinator.com/item?id=31571538
(probably happens with Azure and other smaller platforms as well, e.g. Hetzner, DigitalOcean, Vultr, Scaleway and so on)
That said, most people won't care to put in the work for a multi-cloud/cloud-agnostic setup, since most projects just aren't as important to warrant that much effort. And the ones that are can probably also just talk with the cloud providers through some representative anyways due to spending $$$.I'd argue that it's good to build on common standards, like OCI containers and the wire protocol of some common DBMS like PostgreSQL or MySQL/MariaDB: so that you can replicate certain parts of what a managed service does in a container locally, for development and testing. In most cases it won't matter that the managed cloud offering has some clever engineering underneath it and scales better, as long as you can check whether your SQL executes as it should and none of your seeded/anonymized test data breaks.
I actually had this project with Oracle DB that ran horribly when the instance was remote and you really couldn't do migrations well without breaking things for anyone using it, because the apps using it were written in a way where hundreds if not thousands of SQL queries were done just to load some data to display a page. Which is passable (if you don't care) when the DB is running in the same data centre as the apps, but absolutely horrible when these many smaller SQL queries have the full network round trip between them, especially when the app initializes data sequentially (N+1 problem). Ergo, the only way to work with something like that is to setup a local database (e.g. Oracle XE) and work with it, after importing a baseline migration/doing automated migrations.
The same largely applies to any other DB with sub par application architectures, as well as many other services (e.g. MinIO/Zenko instead of S3, so you don't need to actually upload 100 MB files to test whether attachment logic in your app works, if you can run them locally).
As for software that should be compatible with multiple DBMSes: sometimes it makes sense (e.g. something that could be used by your customers in a self-hosted setup across numerous different setups, like Zabbix or Nextcloud), but most of the time it would negatively impact how easy it is to develop code for your app (e.g. having to rely on ORM and their abstractions like JPQL) and testing everything would usually take more effort.
Re: We reduced our server costs by moving away from AWS
#166Earlier quoted context omitted.
In most businesses I've worked in: developer costs >> infrastructure costs An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense. PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops…
1-2 developer hours?! Anyone looking to hire an ML guy for 250-500$/hr flat fee, hourly rate, no taxes, no healthcare, cancel any time... get in touch! For that money I'll gladly bark like a dog, walk on all fours and fetch your slippers.
In the US salaries are a lot higher so in that context I can see 300-500$/day be realistic.
Really surprised me… Always made me wonder if all that money was being spent well
Re: We reduced our server costs by moving away from AWS
#167I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…
Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance
I'd argue that 90% of the SaaS doesn't have SO's scale. The whole thing would work just fine on a couple of FreeBSD servers running postgres and un-dockerized monolith. Half a rack at most with redundancy and replication.
But, if you've built your whole company around proprietary lamda functions and a vast range of AWS offerings, you're setting up yourself to never get out of the mess.
Re: We reduced our server costs by moving away from AWS
#168They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce yo…
Re: We reduced our server costs by moving away from AWS
#169Earlier quoted context omitted.
> The whole idea that since they have economies of scale, they must pass those savings on to you is a misunderstanding of how companies work. That idea was not expressed in the comment that you replied to. The comment you replied to was in response to a strong claim that ignored the existence of economies of scale, and therefore concluded that it's so obvious that cloud providers must always be more expensive than se…
At this point, it should be pretty clear that self-hosting is cheaper, given all of the examples we have seen. AWS's economies of scale have as much to do with its pricing as the phases of the moon, and citing them as a reason why AWS could be cheaper than self-hosting is pretty ignorant in itself.
Whether economies of scale, or the efficiency of managed services at scale, or use-based loss leaders, there are many ways in which AWS services could be, and often are, cheaper than self-hosting. Of course, not always.
Re: We reduced our server costs by moving away from AWS
#170Earlier quoted context omitted.
Yeah, you need to do calculation of what you are doing right now vs AWS. If you already have a mature team that is doing well optimising the environment, you have stable demand and no need to develop things rapidly then it is very likely that going cloud is poor choice. As to "cloud agnostic", don't believe this bullshit. In my opinion most projects fare much better by just letting go of this "cloud agnostic" and jus…
> As to "cloud agnostic", don't believe this bullshit. > It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS. I agree, I can't tell you how many hours I've wasted trying to keeps something (theoretically) cloud or DBMS agnostic, how many problems it's caused, and at the end of the day we could never "easily" move to a diffe…
The lowest common denominator across clouds is a VM. If you can run your applications and your databases on VMs, you can run them on any cloud AND on-premises. If you can't run CosmosDB or AWS Whatever on a VM, your choice of DBMS/ORM doesn't affect that you're tied to that cloud provider.