Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

131–140 of 261 posts

Re: We 30x'd our Node parallelism

#131

Post by bjacokes »

coddle-hark wrote: post #63
deedubaya wrote: post #36

A good example of avoiding premature optimization. I'd imagine delaying tackling this problem freed them up to tackle problems that impact users.

This only holds if they didn’t pour hours into the original solution. Setting up and managing 4000 node services doesn’t sound like a quick hack.

While we were worried about event loop blockages causing outages, another more subtle problem would have been if event loop blockages doubled our user-facing latency. (If you read the section on latency ratios, you'll see that comparing parallel vs non-parallel workers was the most useful stat in figuring out how effectively we were using the event loop.) It definitely gave us peace-of-mind to know that event loop blockages wouldn't have an effect beyond the requests they're processing.

Honestly, the accounting for which would've been higher impact – investing in parallelism earlier, or adding infrastructure and having more resources to devote to other pressing needs – is difficult to do, even in retrospect. There was surprisingly little effort required to get to 4,000 node containers in an ECS cluster, other than deploy speed issues which we talked about in a previous post [1]. But it's possible this migration process would have been easier if we had done it sooner.

[1] https://blog.plaid.com/how-we-reduced-deployment-times-by-95...

Re: We 30x'd our Node parallelism

#132

Post by markandrewj »

crazygringo wrote: post #129
markandrewj wrote: post #108

Earlier quoted context omitted.

The issue is that many developers that are coming from synchronous programming don't get asynchronous programming. They could both improve the code by not writing blocking code, and also using something like the cluster module ( https://nodejs.org/api/cluster.html ).

I get that. What I don't get is how nobody treats it as an issue when developers coming from Python or Java to C don't get pointers. The assumption is that you learn . But for some reason, people think it's "OK" to not get async, that it's the language's fault rather than the programmer's. That's what I don't understand. It's like a different cultural standard gets applied.

I agree with you, but I think many developers get reluctant to change when they have been doing something one way for a long time, especially if they feel that one way works fine. I can also understand the position, as sometimes it can be fatiguing when technologies are constantly changing. For this project though, if they are actively going to avoid asynchronous programming, they may have been better off choosing a synchronous language.

Re: We 30x'd our Node parallelism

#133

Post by contrahax »

mceachen wrote: post #38

In case anyone else gets excited by JSONStream, know that the package hasn't been updated in over a year, and the GitHub repo was archived by the author with no link to a successor.

I'm maintaining a fork here that incorporates all of the valid open PRs from the original repo + some more updates: https://github.com/contra/JSONStream

It isn't published on NPM (you can use it as a git dependency) but if people are interested I can.

Re: We 30x'd our Node parallelism

#134

Post by a13n »

gameswithgo wrote: post #124
kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

>I think node.js is a great platform, I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node? A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he…

> safety and correctness features

You can achieve safety and correctness features for node via good lint rules and typescript/flow.

Re: We 30x'd our Node parallelism

#136

Post by bdcravens »

FanaHOVA wrote: post #117

$300k is $300k, but they just raised $250M last year, is this a really good use of time for their engineering team? That's a little above ~0.1% of capital.

That's just one of the benefits.

> our system is more robust to increases in external request latencies or spikes in API traffic from our customers

Re: We 30x'd our Node parallelism

#137

Post by bdcravens »

rynop wrote: post #3

I’d be curious to hear your reevaluation of moving this to Lambda after some of the major announcements during re:invent. My guess is some of the reasons you went ECS have been addressed with these announcements. Obviously some of the new features are still preview, but would be interested to hear your analysis none the less.

Oftentimes there's a several month delay from when stuff is announced at re:invent and when it's GA. I don't think anyone would ever make technical decisions based on announcements; they would wait until they could touch it and actually create a proof of concept. In other words, the "analysis" is nonexistent, since there's nothing to analyze.

Re: We 30x'd our Node parallelism

#139

Post by ubu7737 »

bjacokes wrote: post #131
coddle-hark wrote: post #63

Earlier quoted context omitted.

This only holds if they didn’t pour hours into the original solution. Setting up and managing 4000 node services doesn’t sound like a quick hack.

While we were worried about event loop blockages causing outages, another more subtle problem would have been if event loop blockages doubled our user-facing latency. (If you read the section on latency ratios, you'll see that comparing parallel vs non-parallel workers was the most useful stat in figuring out how effectively we were using the event loop.) It definitely gave us peace-of-mind to know that event loop bl…

> But it's possible this migration process would have been easier if we had done it sooner.

What the f*? Of course it would have been easier if you had done it sooner. What you lacked was the willpower from decision-makers who had growth of dollar-signs in their eyes.

You've littered this thread with comments explaining how every move you made was based on ROI. That's the kiss of death for architecture concerns, and bizarrely it puts Node.js on the list of runtimes for data/stream processing backends.

No matter how many times you explain how you made these decisions, I can't help getting the feeling you were wearing horse blinders.

Edit: I find it impossible to imagine that nobody on the engineering team ever shouted, Hey look out! We are basically a Web farm for banking-related requests, this is insane! Surely you've heard from those people and they were let go.

Re: We 30x'd our Node parallelism

#140

Post by calibas »

meritt wrote: post #126
calibas wrote: post #115

Earlier quoted context omitted.

To me, that seems like a case against Javascript. Invalid or broken content should return an error, the parser shouldn't try to "fix" it. And things like data types should be strictly enforced, otherwise you can get unpredictable results, which is especially bad when you're dealing with money transfers.

> Invalid or broken content should return an error, the parser shouldn't try to "fix" it. You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's a…

I'm actually quite experienced with web scraping, mostly using PHP and XPath, but also with Javascript as well as a custom approach written in Rust. I know in detail what an inconsistent mess everything is.

That's why I'm so uncomfortable handling things like bank transfers over such inconsistent, buggy systems, which is what Plaid does. It's not read-only: https://plaid.com/use-cases/consumer-payments/

Not to say I don't trust Plaid, I'm sure they're aware of all this and very careful about how they do things.

Post reply on HN