Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

81–90 of 261 posts

Re: We 30x'd our Node parallelism

#81

Post by Osiris »

winrid wrote: post #61
phoe-krk wrote: post #24

Earlier quoted context omitted.

This is the worst kind of software engineering. There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their co…

Running a Node worker for each thread is standard practice. No different than having a dedicated threadpool for asynchronous programming on the JVM. Yes blocking the event loop is easy. No it's not THAT easy. I've never done it because you think about it while writing code. It's part of the environment. I have had to fix lots of reports etc that try to load up the world and iterate through it in a loop that doesn't y…

> Yes blocking the event loop is easy. No it's not THAT easy. I've never done it because you think about it while writing code.

To be pedantic, all JavaScript functions block the event loop. It's just that the vast majority of functions execute so quickly that the amount of time your function blocks is very short.

I once had a loop that processed tons of data and would block the event loop for 1-3 seconds. I ended up solving it by writing an asynchronous loop which used promises and process.nextTick() to make each iteration a separate execution block on the event loop. But I've only had to do something weird like that once in 10 years of node development.

Re: We 30x'd our Node parallelism

#83

Post by praseodym »

7777fps wrote: post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

Related, from the article:

> We hypothesized that increasing the Node maximum heap size from the default 1.7GB may help. To solve this problem, we started running Node with the max heap size set to 6GB [..], which was an arbitrary higher value that still fit within our EC2 instances.

Sounds like they were utilizing their EC2 instances very poorly. Why not run more workers per instance, or switch to an instance type with less RAM (or more CPUs)?

Re: We 30x'd our Node parallelism

#84

Post by deedubaya »

coddle-hark wrote: post #63
deedubaya wrote: post #36

A good example of avoiding premature optimization. I'd imagine delaying tackling this problem freed them up to tackle problems that impact users.

This only holds if they didn’t pour hours into the original solution. Setting up and managing 4000 node services doesn’t sound like a quick hack.

Writing and maintaining concurrent code for greenfield projects is relatively hard compared to sync code.

Provisioning and deploying with ECS is usually just mouse clicks.

Re: We 30x'd our Node parallelism

#85

Post by reaperducer »

phoe-krk wrote: post #42
earthboundkid wrote: post #31

Earlier quoted context omitted.

Async is just modern cooperative multitasking, and just like the 90s, it's easy to accidentally lock the whole system.

We are no longer in the 90s. The code has increased in volume a hundredfold and it comes from literally everywhere. You can no longer trust everything on your machine or your network to be bug-free or otherwise non-hostile. Creating a system in the 21st century that tries to follow ideals from the 90s gives us the kind of idiotism that we can witness here.

If I thought my bank was running thousands of node containers in parallel to handle transactions, I think I'd look for a new bank.

Re: We 30x'd our Node parallelism

#86

Post by MayeulC »

phoe-krk wrote: post #77
MayeulC wrote: post #72

Earlier quoted context omitted.

> [javascript] makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies. As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red b…

An event loop is basically a single thread that executes functions from a FIFO queue. A function can put itself on a queue by "yielding", which means, allowing other functions to execute. If your function blocks, for example, by performing a wait on something without yielding, then nothing else gets computed because of the wait, but the event queue is occupied since the function has not yielded. This breaks the coope…

All right, thank you. So it's basically the same as any other non-preemptive multitasking design.

If the event loop is part of the Node language/runtime, I can see the case for making it preemptive.

Re: We 30x'd our Node parallelism

#87

Post by neillyons »

MayeulC wrote: post #72
phoe-krk wrote: post #24

Earlier quoted context omitted.

This is the worst kind of software engineering. There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their co…

> [javascript] makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies. As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red b…

In Python, you can execute time.sleep(10_000) in an asyncio program, which means it wont yield back control to the event loop effecting preventing the runtime from doing anything for 10 seconds.

In Javascript world I guess you could do the same by replacing time.sleep with some CPU bound code. eg a big calculation or an infinite for loop.

Re: We 30x'd our Node parallelism

#88

Post by vmarchaud »

I've encounted different issues with NodeJS services in the past (and still do) both with CPU bottleneck and Heap allocations. So i wrote openprofiling-node [0] during this summer to help me profile my apps directly in production and export the result in a S3 bucket. I believe it may help someone else here so i'm posting it

[0]: https://github.com/vmarchaud/openprofiling-node

Re: We 30x'd our Node parallelism

#89

Post by mnutt »

meritt wrote: post #49
Scarbutt wrote: post #33

I don't want to be that guy, but why did they start with nodejs for something like this instead of using the JVM or Go?

My guess is because their system is primarily issuing HTTP requests and extracting data out of responses: html, xml, json, plaintext, etc. Web scraping is a messy business and using a language that allows you to be flexible with string manipulation and types goes a long way toward sanity.

Node is pretty good for managing HTTP requests as long as the responses aren't too large. But parsing data, especially html/xml, is CPU-intensive in node and probably not a great fit.
Post reply on HN