Can you lower our bill now? :)
My one step solution to bill lowering involves "blackmailing your TAM." Have you tried that? ;-)
A million-dollar engineering problem
71–80 of 265 posts
Re: A million-dollar engineering problem
#72Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.
It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…
This article makes a good case for having at least one early team member who knows how to build out physical infrastructure at scale, even if you initially decide on AWS / GCE / Azure:
http://firstround.com/review/the-three-infrastructure-mistak...
Re: A million-dollar engineering problem
#73A great write up. Really wish that they had given some indication about how much the original bill was. Sounds very effective but 1m of 100m isn't that much!
Re: A million-dollar engineering problem
#74> After a three months of focused work, we managed to cut our AWS bill by over one million dollars annually.
That's great, but now I'm wondering how many engineers were paid for three months of full time work to save $1 million/year on AWS, and how much it costs the company to pay those engineers for 3 months.
It's also interesting to wonder about how much growth your startup could have managed if it had had three additional months of work from those engineers.
Re: A million-dollar engineering problem
#75Had an interesting experience when I went to the pricing page for Segment - Had a little sales guy pop up in the lower right corner asking "Would (my enterprise employer name) like to know if you qualify for a volume discount?" I only recently started working (and web browsing) from an enterprise network, so maybe this has been done before, but it's my first time seeing it and I thought it was really interesting. Edi…
Did the box have a "Powered by x" statement anywhere? Would be curious to know if it's Intercom or something else.
Don't see anything like that. Graphically it's seamlessly integrated into the website.
I went back and clicked on it this time and it then says "Our scoring models shows (company name) could be a good fit for our business plan. Want to schedule time to learn more?"
Re: A million-dollar engineering problem
#76Earlier quoted context omitted.
I've always found it interesting that Netflix has not at least tried to go off AWS; if only because Amazon has a competing service with Amazon video. I'm sure its not an easy problem to solve, but Dropbox has gone off AWS as well; it seems like they would be much better served with their own "Video Cloud" with specialized hardware for streaming/processing videos.
Netflix doesn't use AWS for streaming video. https://openconnect.netflix.com/en/
You make a good point that Netflix has a pretty sophisticated system for delivering video (not just served from AWS Cloudfront). However IIUC they still use AWS for preparing video files for their CDN.
Re: A million-dollar engineering problem
#77A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…
I used to do data-center and virtualization consulting- we always designed our enterprise client's systems to handle the baseload in-house and burst to public cloud. Not rock science. We even automatically live-migrated load from and shutdown onsite hypervisors at night when the system had surplus capacity... I think the real issue is that people assume all workloads are public cloud workloads. The bigger and less dy…
This is a great approach. A company I worked at was running mostly on dedicated hardware, and due to the pain of moving jobs around and getting new hardware, we focused hard on optimizing our code to run within our existing capacity whenever we ran into problems. After a couple years of customer growth, I'm sure we would have been running on at least 10x as many machines if we had never done that. We then set up cloud spot instances for when we just needed extra short-term capacity.
Re: A million-dollar engineering problem
#78Earlier quoted context omitted.
I am curious your thoughts on the following. It seems like there is a trough of benefit to AWS. In the mid-size world, especially where computing needs are relatively stable/predictable, you are much better leasing dedicated servers from the usual suspects. Below that size, the linear scaling allows small scrappy startups to really scale/burst with their needs. And at large scale, but below Google/Facebook et al scal…
I agree with a lot of what you're saying-- but that trough is a lot wider than most people think. At the single instance very small scale, it simply doesn't matter all that much. "Our business failed because our infrastructure bill was $400 instead of $150 every month" isn't that common of a story. Base load can be effectively migrated to AWS, depending upon how it's built / structured. It's hard to lose money on inf…
Are bursting and resizing an issue in the trough, though? GP's premise is that these companies have platform and load stability. There can be marginal gains in having a dynamic architecture, sure, but when it also costs "months of engineering," I don't see where the savings would be compelling, or even guaranteed.
Re: A million-dollar engineering problem
#79I've always been set aback by how much AWS servers cost. Maybe I'm just too cheap but you can go to very reputable hosting companies and get things at fractions of the price. For example, if I need 16GB of memory and 4 cores here are my options: * AWS (t2.xlarge) $137.62 * OVH (EG-16) $79 * So You Start/OVH (SYS-IP-1S) $42 * Kimsufi/OVH (KS-3B) $29 * Hetzner (EX41) €46.41 [Lowest cost model is 32GB of ram] * Joe's Da…
AWS is so much more than servers. If you just want a server to run some stuff on, you're right, it's faster and cheaper to buy a dedicated server from a reseller.
With Amazon you have so many options when it comes to storage, compute power, load balancing... They have a huge portfolio of products and you can get everything in one location from one vendor.
Re: A million-dollar engineering problem
#80Earlier quoted context omitted.
The simplest one might be to convince a company to reserve 3 years worth of AWS resources and paying upfront. I am in this situation right now, but and it's a tough pill to swallow. I decided that all of my personal projects will be GCE. It is much more cost efficient already and Google will soon allow me to commit to future usage and pay my commitment as I go (Right now AWS forces you to pay upfront to get the same…
With a couple of exceptions, 3 year RIs are a poor move. You're locking in pricing, and opting out of both newer instance classes and future price reductions during that time period. Generally, they're only useful for "that database we WILL NEVER MOVE," or if you're writing portions of your cloud spend as CapEx and want to amortize depreciation.
I agree on AWS it's a hard move. On GCE, however, you would be buying CPU and memory units, not machines, which to me is much more appealing. Even if the the price drops 50% every year, i would break even.