Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The $10m Engineering Problem

segment.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: The $10m Engineering Problem

#21

Post by wgjordan »

skywhopper wrote: post #7
cle wrote: post #6

> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients. > Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and t…

Consul allows transparent failover to be built in easily. So it can prefer your AZ-local service, but if that becomes unavailable, it can fail over to the next-nearest service, be it in a different AZ or an entirely different region. The direct lookup you describe would not be able to handle failover in an intelligent way. Consul can also provide DNS automatically for your services, route based on network tomography,…

> As for AZ outages... in practice, a cross-AZ system is often just as vulnerable to problems from the outage of a particular AZ, especially if any autoscaling is involved.

If a system is not resilient to an outage of a particular AZ, by definition I would not call it a 'cross-AZ system'. Maybe what you have in mind is systems that in practice _think_ they are cross-AZ resilient but are actually not when you look closer?

Re: The $10m Engineering Problem

#22

Post by cellularmitosis »

patio11 wrote: post #20
stefan_ wrote: post #18

Segment sounds like the kinda business that should presumably just serve up 403 errors for all EU traffic. Data laundering analytics to 300 external tools is shitting on the GDPR. (Remember this when reading the article: all the traffic, all the VMs, all the megadollars spent on AWS here are doing nothing but tunnel ( replicate ) analytics data to third-parties, all of whom would be perfectly happy to receive it dire…

Among other reasons to architect it this way, having the client (web browser) connect to each analytics provider directly pushes the work to the least reliable, most network-constrained, and least manageable node in the network. Segment lets you have the client do de minimis work and have the heavy duty transfer (and retries, etc) happen from somewhere in AWS, where they're not connected over a 3G connection. That is…

not to mention that many businesses have multiple client platforms (web, iOS, android, etc), so implementing anything client-side immediately multiplies the dev spend.

Re: The $10m Engineering Problem

#23

Post by mharroun »

throwaway_bad wrote: post #5

Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!

(Not disagreeing and is somewhat off the article topic to the article but wanted to counter point as engineers love to find reasons to rebuild in new stuff)

So you could also argue for that savings it may not be worth it. 100k is less then 1 engineer in most places (especially including TOTAL hire costs like equipment, office space, benefits, ect).

You can also argue it is much harder to hire golang engineers (or invest time/money into training engineers in go) so that that savings may not be worth it depending on the time it takes to rewrite plus the gamble of failure.

Also you need to ask could those engineers doing the rewrite have been working on up-sell features or other products that could make more money?

Would the investors (especial new growth oriented investors) care more about a increase in margins or a even larger multiple in ARR?

Re: The $10m Engineering Problem

#24

Post by shrubble »

340 VMs is maybe at most, 2 racks of equipment.

So figure 4 racks, 2 racks in each of 2 locations. That's not even 500k in equipment. Telecoms run 'tandem' and that is good enough even for 911 infrastructure.

10gb of decent quality internet at each location is another 2x 5k per month. Power space etc. and remote hands is 2k x 4 racks is 8k per month.

So 500k capex plus 18k per month. And how much are they paying AWS?

Re: The $10m Engineering Problem

#25

Post by dilyevsky »

boulos wrote: post #12

Disclosure: I work on Google Cloud. Awesome writeup! I’ve seen lots of customers do similar “let the packets spray” on both GCP and AWS. Interestingly, it’s one of the reasons I was so excited for our “ILB as next hop” [1] feature. Routing to the Service within the same Zone, unless there’s a failure at which point you wish to go elsewhere in the Region is a common pattern. I’m excited to see where Traffic Director a…

gcp services are always great on marketing booklets but tend to have less than stellar reliability and you gotta read the fine print. A perfect example is ilb only supports max 250 backends and is basically unusable for mutli-regional setups.

Re: The $10m Engineering Problem

#26

Post by imperialdrive »

shrubble wrote: post #24

340 VMs is maybe at most, 2 racks of equipment. So figure 4 racks, 2 racks in each of 2 locations. That's not even 500k in equipment. Telecoms run 'tandem' and that is good enough even for 911 infrastructure. 10gb of decent quality internet at each location is another 2x 5k per month. Power space etc. and remote hands is 2k x 4 racks is 8k per month. So 500k capex plus 18k per month. And how much are they paying AWS?

I gave up trying to spread the good word... it's like gramps telling the youngins to build a nice house somewhere scenic, good bones, raise a family... instead they go live at a WeWork, and rent Ikea furniture by the minute... the tide may turn as money gets less and less cheap.

Still, the engineers gotta feel good about saving all that money, and the environment may feel good about the energy savings too. I hope they get a nice bonus for rolling up their sleeves.

Re: The $10m Engineering Problem

#27

Post by bradleyjg »

kenhwang wrote: post #8
throwaway_bad wrote: post #5

Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!

Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case. I think…

The fully loaded cost of even the most junior dev in a high cost of living area is going to be well above $100k/year. Gross wages are generally only 1/3 to 1/2 the all-in cost.

Re: The $10m Engineering Problem

#28

Post by manigandham »

mnutt wrote: post #11
cle wrote: post #6

> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients. > Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and t…

The EC2 Metadata API isn’t meant for high-throughput calls, so it’s possible to hit rate limits even from moderate polling once you get enough nodes involved.

Why do you have to constantly poll it? It's once on startup to discover the zone it's running in.

Re: The $10m Engineering Problem

#29

Post by manigandham »

It's better to use multiple regions instead of multiple zones in a single region. The costs are very similar (and sometimes even the same) especially with the ridiculous networking fees.

Also object storage is a great way to expand capacity for queueing systems instead of oversized instances. We either write to Kafka or fall back to writing files to S3 across different buckets and providers.

Re: The $10m Engineering Problem

#30

Post by bfung »

shrubble wrote: post #24

340 VMs is maybe at most, 2 racks of equipment. So figure 4 racks, 2 racks in each of 2 locations. That's not even 500k in equipment. Telecoms run 'tandem' and that is good enough even for 911 infrastructure. 10gb of decent quality internet at each location is another 2x 5k per month. Power space etc. and remote hands is 2k x 4 racks is 8k per month. So 500k capex plus 18k per month. And how much are they paying AWS?

It's not quite as simple as that. User boulos's comment below adds more to the estimate given and includes things that tech companies these days don't want to contend with / have flexibility towards.

In general, you're correct, but in colo-skills don't exist with AWS/GCP power users today. Management and skills of tech companies today don't even know where to start for hiring colo-skills. So it's all avoided w/the cloud tax.

Post reply on HN