Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

121–130 of 261 posts

Re: We 30x'd our Node parallelism

#121

Post by bjacokes »

spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

You hit the nail on the head here. When N different API requests simultaneously time out – all because a ramda.uniq call in one of them received an array of 100,000 nested objects – it's easy to make a spot code fix, but harder to systematically prevent it from happening in the future. There aren't really linters for "bad event loop blockage". Code reviews are the main tool we have, but you'd be surprised what sorts of logic can trickily block the event loop. For API reliability and development velocity in the short-term, by far the easiest approach was to throw more infrastructure at the problem.

We do use Go for almost all of our other services, and there are an increasing number of integrations written in Python. But we're still using and investing in our Node integrations code for the foreseeable future, and this was an important step for simplifying our infrastructure.

We certainly hope the tooling and rollout process in the post were instructive for anyone using Node, even if their stacks were pristine from day 1 and never need this sort of complex migration :)

Re: We 30x'd our Node parallelism

#122

Post by megous »

phoe-krk wrote: post #60
james-mcelwain wrote: post #57

Earlier quoted context omitted.

Are you seriously arguing against event loops as a category? Blocking in non-blocking code is going to be an issue whether the language is C or JavaScript (duh).

For these kinds of programming, yes, I argue against it. A single stalling function in Node deadlocks the whole system; a single stalling thread in the C++ model still permits other threads to run. This is a risk that is completely avoidable by not using languages which require event loops at their core.

JavaScript doesn't require event loop design. You can do a PHP like backend design with JS, where each request is handled by a fresh process, and all JS functions block. There's nothing in the language that prevents this.

Some features would become unusable, like Promises and async/await, but those would be worthless in such a design anyway.

Re: We 30x'd our Node parallelism

#123

Post by ubu7737 »

bjacokes wrote: post #110
abalone wrote: post #58

Earlier quoted context omitted.

> In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc. But these are not parallelism problems. These are single threading problems, which the core problem with Node.js, not parallelism in general. Hence I think the question stands: why did you…

It was chosen about 6 years ago when the product was first being developed, so most of us on the engineering team weren't around when the decision was made. The main choice we're making at this point is: what's the impact and ROI of a language migration vs getting Node to work as well as we can?

> what's the impact and ROI of a language migration vs getting Node to work as well as we can?

Hire an architect costs what? Putting the genie back in the bottle was a problem Plaid baked into its early success, which is common with startups hiring engineers with zero architecture knowledge.

Re: We 30x'd our Node parallelism

#124

Post by gameswithgo »

kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

>I think node.js is a great platform,

I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node?

A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he comes up with something -> its dynamically typed -> lets run a huge banking infrastructure on this

wat

Re: We 30x'd our Node parallelism

#125

Post by rdsubhas »

stickfigure wrote: post #74
spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

I'd recommend moving away from Node... Taking a wild guess: Some of their bank integrations probably require browser automation. If you're doing browser automation, the best tool for the job is (currently) Puppeteer, which runs on Node. There are other third-party language bindings for the Chrome dev tools protocol, but Puppeteer is developed by Google as a first-class citizen alongside Chrome.

4000 chrome instances? Probably not. Here I am trying to run 4 chrome instances in parallel in CI without crashing.

Re: We 30x'd our Node parallelism

#126

Post by meritt »

calibas wrote: post #115
meritt wrote: post #97

Earlier quoted context omitted.

I wouldn't characterize it as "better" but specifically easier and more flexible for the people writing and maintaining these scrapers. I'm also speaking more broadly about scripting languages (not javascript specifically) vs the aforementioned JVM or Go, and the ease with which you can deal with inconsistent, frequently changing, and often completely invalid inputs from a wide variety of data sources. Plaid's use ca…

To me, that seems like a case against Javascript. Invalid or broken content should return an error, the parser shouldn't try to "fix" it. And things like data types should be strictly enforced, otherwise you can get unpredictable results, which is especially bad when you're dealing with money transfers.

> Invalid or broken content should return an error, the parser shouldn't try to "fix" it.

You would have a really difficult life in web scraping. You do not have the guarantees of well-formed data. Instead you get HTML with mismatched tags, JSON with newlines in the middle of strings, content that claims it's UTF-8 but upon closer inspection is actually GB2312, pagination endpoints with off-by-one errors, etc. It's an absolute mess and taking the stance of "well, they didn't encode their JSON correctly, so we're not going to operate on their data" isn't a very effective strategy.

> Which is especially bad when you're dealing with money transfers

Afaik Plaid is read-only. They fetch information from financial institutions and make it available through an API.

Re: We 30x'd our Node parallelism

#127

Post by bjacokes »

mnutt wrote: post #50

I’d be curious to hear more about the circumstances that ended up with a blocked runloop. Are there hundreds of junior engineers, or perhaps third parties writing code that you don’t control? I have seen people accidentally write blocking code, but not at such an egregious rate that we couldn’t catch it in code review, or at worst the runloop detector would alert on it in prod and we would roll back the deploy. For i…

Sure, one example I remember off the top of my head is a bank that sometimes returned duplicate transaction data, so an engineer had called ramda.uniq on the transaction array. Transactions are nested objects and slow to compare, so when you find an account with 100,000 transactions... kaboom. Some scenarios are more subtle, but a common theme is that the amount of data in an account can vary by many orders of magnitude.

Re: We 30x'd our Node parallelism

#128

Post by bjacokes »

nosianu wrote: post #48

They write (somewhere in the middle) > Since V8 implements a stop-the-world GC, new tasks will inevitably receive less CPU time, reducing the worker’s throughput But there is this Google blog post vom January 2019: https://v8.dev/blog/trash-talk > Over the past years the V8 garbage collector (GC) has changed a lot. The Orinoco project has taken a sequential, stop-the-world garbage collector and transformed it into a…

Fixed the images about half an hour ago, sorry about this!

Re: We 30x'd our Node parallelism

#129

Post by crazygringo »

markandrewj wrote: post #108
crazygringo wrote: post #99

Earlier quoted context omitted.

Yeah, I don't get it either, at all. The original poster wrote below: > In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc. But it's trivial (a single line) in Node to place breaks in CPU processing to allow the event loop to fire, and as for…

The issue is that many developers that are coming from synchronous programming don't get asynchronous programming. They could both improve the code by not writing blocking code, and also using something like the cluster module ( https://nodejs.org/api/cluster.html ).

I get that.

What I don't get is how nobody treats it as an issue when developers coming from Python or Java to C don't get pointers.

The assumption is that you learn.

But for some reason, people think it's "OK" to not get async, that it's the language's fault rather than the programmer's. That's what I don't understand. It's like a different cultural standard gets applied.

Re: We 30x'd our Node parallelism

#130

Post by fenwick67 »

mceachen wrote: post #38

In case anyone else gets excited by JSONStream, know that the package hasn't been updated in over a year, and the GitHub repo was archived by the author with no link to a successor.

Oboe has a similar API, can't speak for performance though.

http://www.oboejs.com/examples

Post reply on HN