A $10m problem is 1,000,000,000× smaller than a $10M problem.
The $10m Engineering Problem
31–40 of 40 posts
Re: The $10m Engineering Problem
#32Buying rack space at a colo costs money, but if you are spending millions of dollars on AWS you will likely end up spending a few hundred thousand including a salaried sysadmin to manage the hardware. This does mean increased management complexity, so you have to build out an operations team. The total for salaries will be around 400-600k. In the end you will have some setup costs and you will have to choose a subset…
Disclosure: I work on Google Cloud. The blog post doesn’t make it as direct, but one of their biggest costs was for networking between datacenters (Availability Zones in AWS). Most comparisons for “buy a rack at a colo” assume one colo, and a static fleet of hardware. If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at p…
> If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at peak load (“N+1”).
Not true if it's possible to fallback to cloud. That way we can have both high reliability and low cost (other then during outage/maintenance of collocation).
Re: The $10m Engineering Problem
#33Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!
Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case. I think…
We considered a few options before rewriting it. I've got a draft blog post about the process laying around, but haven't gotten around to getting it over the finish line.
We definitely knew the risk going into it. Fortunately, it only took us 2 months to rewrite it. I think our strategy for the rewrite is directly responsible for the speed at which we rewrote it.
Re: The $10m Engineering Problem
#34340 VMs is maybe at most, 2 racks of equipment. So figure 4 racks, 2 racks in each of 2 locations. That's not even 500k in equipment. Telecoms run 'tandem' and that is good enough even for 911 infrastructure. 10gb of decent quality internet at each location is another 2x 5k per month. Power space etc. and remote hands is 2k x 4 racks is 8k per month. So 500k capex plus 18k per month. And how much are they paying AWS?
Righto and while you’re setting all of this up (which usually takes months) your business is going to competition who just spun up a few nodes with a couple lines or terraform (or just autoscaled to demand). Also 10g link is hilarious - you will get x30 that on gcp for this many cores.
Re: The $10m Engineering Problem
#35340 VMs is maybe at most, 2 racks of equipment. So figure 4 racks, 2 racks in each of 2 locations. That's not even 500k in equipment. Telecoms run 'tandem' and that is good enough even for 911 infrastructure. 10gb of decent quality internet at each location is another 2x 5k per month. Power space etc. and remote hands is 2k x 4 racks is 8k per month. So 500k capex plus 18k per month. And how much are they paying AWS?
Righto and while you’re setting all of this up (which usually takes months) your business is going to competition who just spun up a few nodes with a couple lines or terraform (or just autoscaled to demand). Also 10g link is hilarious - you will get x30 that on gcp for this many cores.
[deleted]
Re: The $10m Engineering Problem
#36Earlier quoted context omitted.
Disclosure: I work on Google Cloud. The blog post doesn’t make it as direct, but one of their biggest costs was for networking between datacenters (Availability Zones in AWS). Most comparisons for “buy a rack at a colo” assume one colo, and a static fleet of hardware. If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at p…
> If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at peak load (“N+1”). Not true if it's possible to fallback to cloud. That way we can have both high reliability and low cost (other then during outage/maintenance of collocation).
Hmm. I read the comment as saying “no cloud, because you’ll save so much by just being on-prem”. And I think an “apples-to-apples” comparison requires an N+1 setup including both compute and networking.
Hybrid could be many different setups, but before their “zonal affinity” change it would actually be worse, right? (Egress over Direct Connect is 4x higher than Zone to Zone, while “internet” egress is 8x). What are you assuming for the balance of Compute and Networking across at least three “sites”?
Re: The $10m Engineering Problem
#37Earlier quoted context omitted.
> If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at peak load (“N+1”). Not true if it's possible to fallback to cloud. That way we can have both high reliability and low cost (other then during outage/maintenance of collocation).
Hmm. I read the comment as saying “no cloud, because you’ll save so much by just being on-prem”. And I think an “apples-to-apples” comparison requires an N+1 setup including both compute and networking. Hybrid could be many different setups, but before their “zonal affinity” change it would actually be worse, right? (Egress over Direct Connect is 4x higher than Zone to Zone, while “internet” egress is 8x). What are y…
> Hmm. I read the comment as saying “no cloud, because you’ll save so much by just being on-prem”. And I think an “apples-to-apples” comparison requires an N+1 setup including both compute and networking.
That is valid interpretation. I just wanted to say that is you need high availability it might be cheaper to have one colocation and cloud in standby.
> Hybrid could be many different setups, but before their “zonal affinity” change it would actually be worse, right? (Egress over Direct Connect is 4x higher than Zone to Zone, while “internet” egress is 8x).
Yes, in/out traffic would be one of more problematic points of such setup, but there should be some solutions available (BGP?).
> What are you assuming for the balance of Compute and Networking across at least three “sites”?
Least expensive should be zero compute in cloud unless there is issue with collocation. Depending on specific scenario, some storage/databases would have replication to cloud. I don't know how I would setup networking in such case.
Re: The $10m Engineering Problem
#38Earlier quoted context omitted.
Hmm. I read the comment as saying “no cloud, because you’ll save so much by just being on-prem”. And I think an “apples-to-apples” comparison requires an N+1 setup including both compute and networking. Hybrid could be many different setups, but before their “zonal affinity” change it would actually be worse, right? (Egress over Direct Connect is 4x higher than Zone to Zone, while “internet” egress is 8x). What are y…
> Hmm. I read the comment as saying “no cloud, because you’ll save so much by just being on-prem”. And I think an “apples-to-apples” comparison requires an N+1 setup including both compute and networking. That is valid interpretation. I just wanted to say that is you need high availability it might be cheaper to have one colocation and cloud in standby. > Hybrid could be many different setups, but before their “zonal…
> Least expensive should be zero compute in cloud unless there is issue with collocation.
One more thing: cloud can be great to scale up in peak utility without buying servers that will idle most of the time. It's just that using only cloud might be much more costly, even if it is easier.
Re: The $10m Engineering Problem
#39Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!
Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case. I think…
Yeah, but junor from HCoL is not the only option, so why would you consider the most expensive one?
Re: The $10m Engineering Problem
#40> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients. > Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and t…
The EC2 Metadata API isn’t meant for high-throughput calls, so it’s possible to hit rate limits even from moderate polling once you get enough nodes involved.
The AZ for an instance is fixed. Check it at startup time, and cache it in-memory.