The $10m Engineering Problem • Hacker News Board

Re: The $10m Engineering Problem

#11

Post by mnutt » Sat, Oct 19, 2019, 10:19 PM UTC

cle wrote: post #6

> Then when a reader connects, instead of connecting directly to the nsqlookupd discovery service, the reader connects to a proxy. The proxy has two jobs. One is to cache lookup requests, but the other is to return only in-zone nsqd instances for zone-aware clients. > Our forwarders that read from NSQ are then configured as one of these zone-aware clients. We run three copies of the service (one for each zone), and t…

The EC2 Metadata API isn’t meant for high-throughput calls, so it’s possible to hit rate limits even from moderate polling once you get enough nodes involved.

Re: The $10m Engineering Problem

#12

Post by boulos » Sat, Oct 19, 2019, 10:30 PM UTC

Disclosure: I work on Google Cloud.

Awesome writeup! I’ve seen lots of customers do similar “let the packets spray” on both GCP and AWS. Interestingly, it’s one of the reasons I was so excited for our “ILB as next hop” [1] feature. Routing to the Service within the same Zone, unless there’s a failure at which point you wish to go elsewhere in the Region is a common pattern. I’m excited to see where Traffic Director and similar service mesh patterns lead in this space. Having everyone need to roll this by hand, seems needlessly redundant.

As an aside, another personal surprise when I got into an argument over Zone-to-Zone pricing: GCE only charges for one-way while AWS charges for send and receive. We should clearly do better marketing :).

[1] https://cloud.google.com/load-balancing/docs/internal/ilb-ne...

Re: The $10m Engineering Problem

#13

Post by ltbarcly3 » Sat, Oct 19, 2019, 11:10 PM UTC

Buying rack space at a colo costs money, but if you are spending millions of dollars on AWS you will likely end up spending a few hundred thousand including a salaried sysadmin to manage the hardware.

This does mean increased management complexity, so you have to build out an operations team. The total for salaries will be around 400-600k.

In the end you will have some setup costs and you will have to choose a subset of the features AWS offers, but you'll save millions of dollars per year and have much better performing hardware and much, much more flexibility.

AWS is extremely expensive.

Re: The $10m Engineering Problem

#14

Post by arcticbull » Sat, Oct 19, 2019, 11:10 PM UTC

kenhwang wrote: post #8

throwaway_bad wrote: post #5

Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!

Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case. I think…

Thats all true of course assuming it'll take (?) a full year to do.

Re: The $10m Engineering Problem

#15

Post by ltbarcly3 » Sat, Oct 19, 2019, 11:11 PM UTC

kenhwang wrote: post #8

throwaway_bad wrote: post #5

Always interesting to see the scale you have to hit before rewriting from one language to another saves money (relative to engineering cost). With node.js: 800 containers, with each container processing 250 messages per second With golang: 340 containers, with each container processing 650 messages per second Say each one of those containers cost $0.02/hr then that's order of $100k/year saved!

Considering a typical HCoL junior dev costs about ~100k/yr, if you can have one junior dev rewrite your entire codebase in a year, you'll breakeven in cost after 2 years. Considering a senior dev costs 2-3x that amount per year, as soon as you have one of those involved for an entire year (odds are, if it's business critical software you will), your breakeven point comes out to just under a decade worst case. I think…

If you have one junior dev rewrite your entire codebase they'll never finish and the quality of the rewrite will be terrible. It's right in the definition of 'junior'.

Re: The $10m Engineering Problem

#16

Post by boulos » Sat, Oct 19, 2019, 11:33 PM UTC

ltbarcly3 wrote: post #13

Buying rack space at a colo costs money, but if you are spending millions of dollars on AWS you will likely end up spending a few hundred thousand including a salaried sysadmin to manage the hardware. This does mean increased management complexity, so you have to build out an operations team. The total for salaries will be around 400-600k. In the end you will have some setup costs and you will have to choose a subset…

Disclosure: I work on Google Cloud.

The blog post doesn’t make it as direct, but one of their biggest costs was for networking between datacenters (Availability Zones in AWS). Most comparisons for “buy a rack at a colo” assume one colo, and a static fleet of hardware.

If you wanted to compare apples-to-apples, you’d need to have (at least) three nearby colos with enough capacity to handle one going down entirely at peak load (“N+1”). Leased lines in a metro area aren’t actually all that expensive, but like the compute, you also need to purchase that with failure in mind.

tl;dr: Maybe, but the analysis needs to assume the same(ish) reliability outcome. Otherwise, they could have avoided lots of cost by just running in a single Zone.

Re: The $10m Engineering Problem

#17

Post by lifeisstillgood » Sat, Oct 19, 2019, 11:49 PM UTC

>>> As a concrete example: a single Salesforce server supports thousands or millions of users, since each user generates a handful of requests per second. A single Segment container, on the other hand, has to process thousands of messages per second–all of which may come from a single customer.

This sounds like the basic problem with Big Data and selling advertising as a business model ... that eventually even bits aren't free.

I can see how it happens - but I think any business that has as its core ship everything to our servers in San Francisco is just badly architected - and if that's your business model you have a bad business model.

no particular comment on segment but a general thought - perhaps most of the business models today are not very good ones

(I seems to remember a rap lyric start up that spun up a new single threaded Ruby on Raiks instance for the most trivial request increases)

Re: The $10m Engineering Problem

#18

Post by stefan_ » Sat, Oct 19, 2019, 11:54 PM UTC

Segment sounds like the kinda business that should presumably just serve up 403 errors for all EU traffic. Data laundering analytics to 300 external tools is shitting on the GDPR.

(Remember this when reading the article: all the traffic, all the VMs, all the megadollars spent on AWS here are doing nothing but tunnel (replicate) analytics data to third-parties, all of whom would be perfectly happy to receive it directly. It is the definition of waste.)

Re: The $10m Engineering Problem

#19

Post by elesbao » Sun, Oct 20, 2019, 12:04 AM UTC

this looks as much as an engineering problem as a learning problem (how to build systems for cloud) and management (how to track and establish better quality in the whole product lifecycle). nice that they are learning their stuff still and having fun.

Re: The $10m Engineering Problem

#20

Post by patio11 » Sun, Oct 20, 2019, 12:16 AM UTC

stefan_ wrote: post #18

Segment sounds like the kinda business that should presumably just serve up 403 errors for all EU traffic. Data laundering analytics to 300 external tools is shitting on the GDPR. (Remember this when reading the article: all the traffic, all the VMs, all the megadollars spent on AWS here are doing nothing but tunnel ( replicate ) analytics data to third-parties, all of whom would be perfectly happy to receive it dire…

Among other reasons to architect it this way, having the client (web browser) connect to each analytics provider directly pushes the work to the least reliable, most network-constrained, and least manageable node in the network. Segment lets you have the client do de minimis work and have the heavy duty transfer (and retries, etc) happen from somewhere in AWS, where they're not connected over a 3G connection. That isn't waste, contingent on the company or the user getting value out of analytics and analytics-driven decisionmaking, which is quite plausible.