Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

71–80 of 261 posts

Re: We 30x'd our Node parallelism

#71

Post by winrid »

phoe-krk wrote: post #42
earthboundkid wrote: post #31

Earlier quoted context omitted.

Async is just modern cooperative multitasking, and just like the 90s, it's easy to accidentally lock the whole system.

We are no longer in the 90s. The code has increased in volume a hundredfold and it comes from literally everywhere. You can no longer trust everything on your machine or your network to be bug-free or otherwise non-hostile. Creating a system in the 21st century that tries to follow ideals from the 90s gives us the kind of idiotism that we can witness here.

I mean, if you write slow code in a high throughput environment you'll just kill the CPU from context switching between threads instead.

It doesn't matter if it's in an event loop or thread per request. Architect things correctly.

Re: We 30x'd our Node parallelism

#72

Post by MayeulC »

phoe-krk wrote: post #24
nicoburns wrote: post #9

Earlier quoted context omitted.

> We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. So they fixed the issue that some requests blocked... by making all requests blocking.

This is the worst kind of software engineering. There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their co…

> [javascript] makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies.

As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red button.

Re: We 30x'd our Node parallelism

#73

Post by eshyong »

phoe-krk wrote: post #55
eshyong wrote: post #46

Earlier quoted context omitted.

I mean, running multiple node runtimes (aka multiprocessing) actually sounds like a reasonable compromise for parallelism. That's the standard solution for dynamic languages without great multithreading support. If you needed great multithreading support then Node probably wasn't the right choice for you in the first place, but for most applications, it's probably fine. However, running multiple containers for parall…

Yet this is the case mentioned in the article. > We were running 4,000 Node containers (or "workers") for our bank integration service.

Yes, I agree. I'm arguing against your characterization of node as a poor runtime. For many line-of-business applications node is a fine choice. However, plaid has to integrate with many banks which only expose web pages, not APIs, so I'm guessing that they have to do a non-trivial amount of CPU work to scrape and process HTML responses. For this, it may not be such a good choice.

All I'm saying is that the choice of language is (usually) not the issue. Poor architecture design causes a lot more problems than whether you choose python/java/ruby/node for your webapp.

Re: We 30x'd our Node parallelism

#74

Post by stickfigure »

spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

I'd recommend moving away from Node...

Taking a wild guess: Some of their bank integrations probably require browser automation. If you're doing browser automation, the best tool for the job is (currently) Puppeteer, which runs on Node. There are other third-party language bindings for the Chrome dev tools protocol, but Puppeteer is developed by Google as a first-class citizen alongside Chrome.

Re: We 30x'd our Node parallelism

#75

Post by [deleted] »

spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

[deleted]

Re: We 30x'd our Node parallelism

#76

Post by calibas »

meritt wrote: post #49
Scarbutt wrote: post #33

I don't want to be that guy, but why did they start with nodejs for something like this instead of using the JVM or Go?

My guess is because their system is primarily issuing HTTP requests and extracting data out of responses: html, xml, json, plaintext, etc. Web scraping is a messy business and using a language that allows you to be flexible with string manipulation and types goes a long way toward sanity.

How is Javascript better at string manipulation? I've never encountered anything special there that I can't do in just about every other language. Javascript just has more helper functions out of the box.

Re: We 30x'd our Node parallelism

#77

Post by phoe-krk »

MayeulC wrote: post #72
phoe-krk wrote: post #24

Earlier quoted context omitted.

This is the worst kind of software engineering. There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their co…

> [javascript] makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies. As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red b…

An event loop is basically a single thread that executes functions from a FIFO queue. A function can put itself on a queue by "yielding", which means, allowing other functions to execute.

If your function blocks, for example, by performing a wait on something without yielding, then nothing else gets computed because of the wait, but the event queue is occupied since the function has not yielded. This breaks the cooperative part of the cooperative multithreading mechanism of Node.

Re: We 30x'd our Node parallelism

#78

Post by awinter-py »

Compared to a compiled language, node / JIT langs make it difficult to know what will be fast in prod.

V8 JIT means that things like order of keys in an object or number of different calls to a function might affect whether your function gets optimized.

And there's no easy way to find out if a JS function is falling back to slow mode or to tell the buildsystem 'this is a hot path, don't let me write code that deopts this call'.

Re: We 30x'd our Node parallelism

#79

Post by paulddraper »

stickfigure wrote: post #74
spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

I'd recommend moving away from Node... Taking a wild guess: Some of their bank integrations probably require browser automation. If you're doing browser automation, the best tool for the job is (currently) Puppeteer, which runs on Node. There are other third-party language bindings for the Chrome dev tools protocol, but Puppeteer is developed by Google as a first-class citizen alongside Chrome.

I think that overemphasizes Pupeteer itself.

It's really just bindings for the dev tools protocol.

Half the GitHub issues result in "well the protocol requires X and we can't change that".

Pupeteer is popular because it's web automation protocol bindings for a web language, not because it a sophisticated layer or does very much.

There are literally dozens of language bindings for the protocol. [1] Some are quite good and widely used, for example chromedp (Go bindings). [2]

[1] https://github.com/ChromeDevTools/awesome-chrome-devtools#pr...

[2] https://github.com/chromedp/chromedp

Re: We 30x'd our Node parallelism

#80

Post by mnutt »

bjacokes wrote: post #51
rauchp wrote: post #8

That was an interesting read, thanks for linking to it. It's hard finding articles online discussing Node and performance, most people just dismiss it as an unviable option due to scale and speed concerns. 30x really is quite the jump though. > Each Node worker runs a gRPC server Not going to lie, this kind of surprised me. When I think of a Node backend I think of ExpressJS. Not because I think Express is better, bu…

Our integrations are primarily written in Node, which was the original language used for everything at Plaid. Almost all of those original services (except for integrations) have been migrated to Go or Python at this point. We've standardized on gRPC as our wire format, so we stayed consistent and used gRPC in Node. With perfect hindsight, it's a fair point that all the pros and cons could net out to another language…

I've been hoping that the Cloudflare folks will open source parts of their Workers; they seem to have figured out a secure, performant way to run untrusted javascript at scale.
Post reply on HN