The $10m Engineering Problem
segment.com
The $10m Engineering Problem
1–10 of 40 posts
Re: The $10m Engineering Problem
#2Its a bit silly that moving something from one app to another is so complicated.
Re: The $10m Engineering Problem
#3Does anybody else find it funny when companies talk about typically private metrics (like gross margins) very publicly.
Re: The $10m Engineering Problem
#4This is awesome. I'd love to see more blog posts tying business value to engineering problems, finding ways to measure before/after outcomes, and then sharing the engineering details. What a great post.
Re: The $10m Engineering Problem
#5Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost).
With node.js: 800 containers, with each container processing 250 messages per second
With golang: 340 containers, with each container processing 650 messages per second
Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!
Re: The $10m Engineering Problem
#6> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients.
> Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and then have each send traffic only to the service in its zone.
Isn't this the default behavior of ELB/NLB to begin with? Why not just configure the zone-aware clients to call zonal LBs, instead of hosting your own LB? Same with Consul. I'm not understanding what benefit Segment gets from using Consul vs. calling EC2 Metadata API to discover the AZ and then calling the appropriate zonal LB endpoint...that's not hard to do and avoids many extra dimensions of operational complexity.
It's also unclear to me how all this migration to intra-AZ routing affects Segment's resilience to AZ outages.
Re: The $10m Engineering Problem
#7> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients. > Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and t…
Consul allows transparent failover to be built in easily. So it can prefer your AZ-local service, but if that becomes unavailable, it can fail over to the next-nearest service, be it in a different AZ or an entirely different region. The direct lookup you describe would not be able to handle failover in an intelligent way. Consul can also provide DNS automatically for your services, route based on network tomography, and the latest versions can provide automatic mTLS between services, and descriptive network security rules. Not to mention providing a handy place to store config state and send events.
Beyond that, ELBs have a significant cost if you are running multiple for each internal service you might have, and the API is slow and cumbersome compared to dealing with Consul's service-centric API. From an operations POV, Consul's ACL system is also a lot more flexible than what AWS IAM can provide. So you can be sure your services are limited in what they can claim to be and what gets set up on their behalf. Whereas if you want to automate creation and configuration of ELBs, you are going to have to either grant more access than you really want or you'll have to abstract that behind another service that you have to write.
As for AZ outages... in practice, a cross-AZ system is often just as vulnerable to problems from the outage of a particular AZ, especially if any autoscaling is involved. AWS's tools around this are severely lacking, despite what they tell us about resiliency best practices. But it all depends on the architecture and mostly the data layer.
Re: The $10m Engineering Problem
#8Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!
Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case.
I think that just illustrates how risky rewrites are. Very few companies at that scale can just rewrite everything in that timeframe using that little resources. Many companies don't even have codebases that will survive a decade.
Re: The $10m Engineering Problem
#9They have a Gross Margin Team??? Wow
Re: The $10m Engineering Problem
#10This was a great engineering blog post in that it did a good job in describing, in detail, a large overlying problem (excess intra AZ bandwidth), its impact on margin (20%) and concrete steps to measure the problem and solution. This is exactly the type of communication we should be able to use as an example of the outsize effect engineering can have on the long term value of the company -- if there's anyway that margin increase can in some way be turned into CAGR, these increased margins could double the company's valuation in 4 years (in an ideal world, of course).
Very cool.