Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

21–30 of 261 posts

Re: We 30x'd our Node parallelism

#21
post #10

"Only 10% of Plaid's data pulls involve a user who is present" Since they provide an API, it seems like some of the calls where they think a user isn't present might actually have one present.

The other 90% are not triggered by the API, they are "periodic transaction updates" - presumably they refresh once a day or something.

Yeah, I read that, but it's not clear exactly what those calls are. It sorta sounds like making assumptions on how their users are using the API.

In fact, it sounds like they think "linking an account" is the only "user present" API call:

"Only 10% of Plaid's data pulls involve a user who is present and linking their account to an app"

Re: We 30x'd our Node parallelism

#22
post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

The way it happens always: many people working to a solution, not agreeing on one and then comprising on something in the middle, even if it makes no sense.

Re: We 30x'd our Node parallelism

#23
post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

I wonder what percentage of the massive compute power of huge cloud data centers is spent just chugging away on ugly clunky hacks to run bad code?

Re: We 30x'd our Node parallelism

#24
post #5

Earlier quoted context omitted.

> I thought one of the key selling points with Node was an fully async standard library, enabling better scaling in process. We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. The fact that Node ships a fully asynchronous standard library doesn't in any way fix the fact that Node is a runtime for a language that itself is a mi…

> We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. So they fixed the issue that some requests blocked... by making all requests blocking.

This is the worst kind of software engineering.

There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their code in no way presses that red button on purpose or even by accident.

These people insist that it is impossible to program normally and in a language that is actually sane and does not advertise obvious and gaping design mistakes as "features of the language". These people advertise the analogue of Python's Global Interpreter Lock as the core foundation of their language.

These people advertise Node and the language it implements as practical for implementing multithreaded applications. Posts such as this show what sort of bullshit it is; it is only practical to use Node for parallelism if each single Node instance is only ever run single-threaded. You don't parallelize by running multiple threads, you parallelize by running multiple Node runtimes.

This is no longer an act against productivity or usability. This is simply insane and shows one of the most basic things that are wrong about Node's language and approach. It is impossible to write a multithreaded program if your language of choice makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies.

Re: We 30x'd our Node parallelism

#25
post #16

Earlier quoted context omitted.

You can use something like LMDB on every language.

Ah, yeah. I suppose that would mean you need a fast node.js serializer. Apcu uses its own serializer that is fast-ish.

You can use something like FlattBuffers to have non-copying reads.

Re: We 30x'd our Node parallelism

#26
It's not clear from the article why they were only able to run one request per node process, but that alone would make it questionable why use Node at all then. The entire point of the environment has been nixed. The article is quite confounding to understand how they arrived at that point in the first place.

Re: We 30x'd our Node parallelism

#27
post #5
post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

> I thought one of the key selling points with Node was an fully async standard library, enabling better scaling in process. We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. The fact that Node ships a fully asynchronous standard library doesn't in any way fix the fact that Node is a runtime for a language that itself is a mi…

> trivially blocked by very simple programmer errors

Can you give an example please?

I think it's much easier to block a thread with C#'s async programming model than node's...

Re: We 30x'd our Node parallelism

#28
post #5

Earlier quoted context omitted.

> I thought one of the key selling points with Node was an fully async standard library, enabling better scaling in process. We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. The fact that Node ships a fully asynchronous standard library doesn't in any way fix the fact that Node is a runtime for a language that itself is a mi…

> We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here. So they fixed the issue that some requests blocked... by making all requests blocking.

It's simple but effective.

Re: We 30x'd our Node parallelism

#29
post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

Pretty sure apache + cgi would scale better :)

Re: We 30x'd our Node parallelism

#30
post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

There are a couple of reasons that the legacy scaling model was viable for us. As mentioned in the post, only 1/10 of our traffic was from the API, which gave us a roundabout way to scale by diverting resources. And it's only viable to use this model of scaling when the business value of a request is high – we were originally quite happy to spin up more containers when we reached our scaling limit. That's the pragmatic reason why we were processing one request per container.

In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc.

Post reply on HN