I wish the article went into detail about what hardware they used for each server, what was their disaster mitigation plan, and other considerations that you don't need to worry about with paying for a cloud provider.
We reduced our server costs by moving away from AWS
291–300 of 355 posts
Re: We reduced our server costs by moving away from AWS
#292I've said this a hundred times and it seems not loud enough. AWS is not cheap because of your server costs. AWS is cheap because of elasticity, velocity (opportunity cost of next feature), and reduced maintenance hours. "The cloud" was never (afaik) was about getting a cheaper VPS. It was about being able to get them on demand, give them back on demand, and generally not have to maintain anything besides your code (a…
Prerender as I can see is founded by Hungarian and most employees are Hungarian. Even if he hired 2 FTEs in Hungary to maintain (which I doubt), it would eat 200k at most (probably much less), so they still saved 600k. For 800k he could probably hire 10 more people, to improve development, sales, marketing, support, and that would be better investment instead of burning money on AWS.
I’m from Hungary, and I have to tell you, you are WAY off with those salaries.
As a senior software engineer, you can make maybe $35k a year, before taxes, if you are good. You can make it $50k if you are very good.
2 years ago I was making $23k (yearly, before taxes), before I moved to Canada and started working at Amazon for $170k usd.
Europe, especially the eastern parts of europe has an extremely cheap workforce.
Re: We reduced our server costs by moving away from AWS
#293Earlier quoted context omitted.
Brazil?
This is a reference to the film Brazil[0], which centers on a labyrinthine bureaucracy [0] https://www.imdb.com/title/tt0088846/
. . . and ducting
Re: We reduced our server costs by moving away from AWS
#294Earlier quoted context omitted.
One time I was doing research on some really cheap data centers in basically 3rd world countries. Just out of curiosity to see how cheap it could really get. One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture? I th…
Share the company name / picture please :)
It was years back, I don't think I could find it again. I wish I saved it. Finding the company in the first place was a rabbit hole project. If my memory is correct is was in Brazil.
I really should start a blog for some of the weird research I do sometimes.
Re: We reduced our server costs by moving away from AWS
#295Earlier quoted context omitted.
The value prop of AWS/GCP etc go beyond bare metal vs VMs. Running HA databases (even if you are a startup, you need this), centralized logging, secrets manager, KMS, making sure disks are encrypted, something like pub/sub as a messaging backbone for your application, load balancers... the list goes on. Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the servi…
Yet most of these aren't needed for small crews... do I care that my DB is not HA when I have 1 RPS? You said yes, but I disagree. Do I need centralized logging if I have a few servers? It's already centralized for free. It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services. People pretend that AWS doesn't cost engineers to run it,…
YMMV, I run a startup that serves critical customer use cases, and from day 1 I had to care that my DB was HA, backed up, and could be restored. Actually had a case early on where we had to test this because I accidentally deleted customer data and had to restore - took all of one click, with no scripts, no prior investment into backup management etc. We also had to go through infosec reviews from day 1, and many of these are a must have. Doesn't matter if it's a small crew or large. Our customers care.
3 years fast forward with hundreds of customers, and we haven't had a need to hire someone full time to "manage" these services. This may/will change down the line, but I couldn't have built my business without one of these cloud vendors.
Edit: I also know the pain of building (and managing) this on bare metal (last company did that) - and it was just apache/php/mysql on bare metal. It was a mess
Re: We reduced our server costs by moving away from AWS
#296Earlier quoted context omitted.
Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…
> have to hire But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away. There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people. At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scal…
Agreed. I was doing some unrelated research yesterday and stumbled upon the fact that apparently (in Australia) 41% of companies see an increase in IT staff after adopting cloud services. [1]
I thought this was pretty weird since the original value proposition (AFAIR) was to reduce costs/head count. But everywhere I’ve worked that used AWS, had specialists employed to manage AWS. And I think the value proposition is more that it’s easier to find people who know AWS - because there are training providers and certifications - than people who know how to do everything themselves.
Once you get to a certain size I think you can attract a team who can build out rather than buy in, and in so doing reduce costs.
[1] https://www2.deloitte.com/au/en/pages/economics/articles/eco...
Re: We reduced our server costs by moving away from AWS
#297Earlier quoted context omitted.
How did you get them to approve a large amount of Cloud instances / dedicated servers? I heard they are very stubborn to increase the per user limit of cloud instances. Also how did you deal with S3? Did you switch to another provider ? Like B2?
We haven't run into the number of servers issue. For S3, we've switched to Wasabi which very nicely uses the identical API.
Honestly Wasabi's advantage is free egress, but with b2 + cloudflare gives a better billing system without the three month object retention thing and a CDN with no egress limit.
Re: We reduced our server costs by moving away from AWS
#298Earlier quoted context omitted.
Most of the scripts I was familiar with from that time were one-offs. The code wasn't very reusable. They were automating a task on one machine. Today's "DevOps" are automating things across N machines. It is a matter of scale.
> They were automating a task on one machine. Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc. That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
Exactly - I used to be the sysadm on a single machine in a medical laboratory in the 90s. It was a DG Aviion and with the storage unit took up a whole room. Maybe 300 people used it simultaneously.
The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 months or so - on cartridge tape.
It was about need, not competence.
Re: We reduced our server costs by moving away from AWS
#299Hi! I’m Todd, the solopreneur founder of Prerender.io and I created that $1,000,000/year AWS bill. I sold Prerender.io to Saas.group in 2020 and the new team has done an incredible job growing and changing Prerender since I left.
$1M per year bill is a lot, but the Prerender back end is extremely write-heavy. It’s constantly loading URLs in Chrome in order to update the cached HTML so that the HTML is ready for sub-second serving to Google and other crawlers.
Being a solo founder with a profitable product that was growing organically every month, I really didn’t have the time to personally embark on a big server migration with a bunch of unknown risks (since I had never run any bare metal servers before). So the architecture was set early on and AWS allowed me the flexibility to continue to scale while I focused on the rest of business.
Just for a little more context on what was part of that $1M bill, I was running 1,000+ ec2 spot instances running Chrome browsers (phantomjs in the early days). I forget which instance type but I generally tried to scale horizontally with more smaller instance sizes for a few different reasons. Those servers, the rest of the infrastructure around rendering and saving all the HTML, and some data costs ended up being a little more than 50% the bill. Running websites through Chrome at scale is not cheap!
I had something like 20 Postgres databases on RDS used for different shards containing URL metadata, like last recache date. It was so write heavy that I had to really shard the databases. For a while I had one single shard and I eventually ran into the postgres transaction ID wraparound failure. That was not fun so I definitely over provisioned RDS shards in the future to prevent that from happening again. I think RDS costs were like 10%.
All of the HTML was stored in s3 and the number of GET requests wasn’t too crazy but being so write heavy on PUT requests for recaching HTML, with a decent sized chunk of data, the servers to serve customer requests, and data-our from our public endpoint, that was probably 30%.
There were a few other things like SQS for populating recache queues, elasticache, etc.
I never bought reserved instances and I figured the new team would go down that route but they blew me away with what they were able to do with bare metal servers. So kudos to the current Prerender team for doing such great work! Maybe that helps provide a little more context for the great comments I’m seeing here.
Re: We reduced our server costs by moving away from AWS
#300Earlier quoted context omitted.
> They were automating a task on one machine. Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc. That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.
Exactly - I used to be the sysadm on a single machine in a medical laboratory in the 90s. It was a DG Aviion and with the storage unit took up a whole room. Maybe 300 people used it simultaneously. The idea that I would need to automate the installation of an OS and applications on a fleet of machines was never contemplated because it didn’t make sense. I had one machine and OS upgrades arrived in the post every 6 mo…
The places I'm talking about were mid 90's, dozens of AlphaServers, Sun Sparcs, HP/UX systems, IBM AIX machines, etc. It was more than a single machine, but less than 50. There was shocking little automation. Everything was a pet, with crazy NFS mounts all over the place, custom scripts in /usr/local/etc not in version control, etc. If there was ever a power loss, it was a pain getting everything back up.