Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

111–120 of 261 posts

Re: We 30x'd our Node parallelism

#111

Post by jrockway »

PixyMisa wrote: post #82

Nobody involved in this project should be allowed to ever be in the same room as a computer again.

Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000.

Seems like everything went right to me.

I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "we rewrote it in go and in the rewrite introduced 87 new bugs while fixing 42 old bugs". They engineered a solution, built from good investment in infrastructure, rather than ninja-ing a hack. That, to me, is a very good thing.

A lot of people seem deeply upset that Node was involved, but I think that's a red herring. The problem they had -- allocate a large chunk of memory, keep a reference to it while it is slowly sent to another server, free memory -- is going to happen in any language. (I don't super agree with their solution of "make the server faster" because one day it's going to be slow for some other reason and this problem will crop up again. Instead they probably just need a fixed amount of memory to dedicate to this process and to drop the debug payload when the buffer is full. Or just put it in the request path if it's crucial that it be produced every time no matter what. At least that will apply backpressure to calling services, pop the circuit breaker, and redirect requests to a region where S3 isn't broken. But I don't think the debug information is THAT important ;)

Re: We 30x'd our Node parallelism

#112

Post by inglor »

NathanKP wrote: post #103
kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

This is why I recommend that anyone running Node in production use a tracing tool like New Relic. It's super easy to see what is blocking the event loop. Just choose a duration (say 10ms) and look for any execution spans that are longer than that duration. Ideally you want to be yielding back to the event loop at least every 1 ms. Anything that takes too long without yielding will show up as a latency delay before yo…

Elastic APM is better than New Relic in how it traces node and it is completely free and open source (you can use a cloud product).

Disclosure and bias: I work on Node core and always hear ranting about incorrect usage in async_hooks in anyone but Elastic APM in core meetings. I used both products and have no affiliation to other companies.

Re: We 30x'd our Node parallelism

#113

Post by tluyben2 »

7777fps wrote: post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

> I can't be the only person who reads stories like this and wonders how they arrived at that solution in the first place?

No you are not. I wonder which CTO would allow this; like everyone here, the exact case is not really clear (or at least why this solution is a great solution for it), but this sounds like a weird solution (and expensive) to some issue. I really don't understand these 'solutions' and I am almost 100% sure I (with a team! but the point that this is not the best solution for the problem) can whip up something far simpler and more efficient for this problem. But ofcourse there are problems that might fit?

Re: We 30x'd our Node parallelism

#114

Post by NathanKP »

inglor wrote: post #112
NathanKP wrote: post #103

Earlier quoted context omitted.

This is why I recommend that anyone running Node in production use a tracing tool like New Relic. It's super easy to see what is blocking the event loop. Just choose a duration (say 10ms) and look for any execution spans that are longer than that duration. Ideally you want to be yielding back to the event loop at least every 1 ms. Anything that takes too long without yielding will show up as a latency delay before yo…

Elastic APM is better than New Relic in how it traces node and it is completely free and open source (you can use a cloud product). Disclosure and bias: I work on Node core and always hear ranting about incorrect usage in async_hooks in anyone but Elastic APM in core meetings. I used both products and have no affiliation to other companies.

Interesting! I've had too many bad experiences over the years with Elasticsearch and ELK stack so I usually avoid Elasticsearch based products like the plague. Maybe if someone else runs it for me though...

Do you know what they are doing differently with async_hooks in Elastic APM?

Re: We 30x'd our Node parallelism

#115

Post by calibas »

meritt wrote: post #97
calibas wrote: post #76

Earlier quoted context omitted.

How is Javascript better at string manipulation? I've never encountered anything special there that I can't do in just about every other language. Javascript just has more helper functions out of the box.

I wouldn't characterize it as "better" but specifically easier and more flexible for the people writing and maintaining these scrapers. I'm also speaking more broadly about scripting languages (not javascript specifically) vs the aforementioned JVM or Go, and the ease with which you can deal with inconsistent, frequently changing, and often completely invalid inputs from a wide variety of data sources. Plaid's use ca…

To me, that seems like a case against Javascript. Invalid or broken content should return an error, the parser shouldn't try to "fix" it.

And things like data types should be strictly enforced, otherwise you can get unpredictable results, which is especially bad when you're dealing with money transfers.

Re: We 30x'd our Node parallelism

#116

Post by PixyMisa »

jrockway wrote: post #111
PixyMisa wrote: post #82

Nobody involved in this project should be allowed to ever be in the same room as a computer again.

Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "w…

To save $300,000 they first needed to waste $300,000 by reinventing a problem that was solved in 1967.

Re: We 30x'd our Node parallelism

#118

Post by [deleted] »

crazygringo wrote: post #99
7777fps wrote: post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

Yeah, I don't get it either, at all. The original poster wrote below: > In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc. But it's trivial (a single line) in Node to place breaks in CPU processing to allow the event loop to fire, and as for…

[deleted]

Re: We 30x'd our Node parallelism

#120

Post by ubu7737 »

jrockway wrote: post #111
PixyMisa wrote: post #82

Nobody involved in this project should be allowed to ever be in the same room as a computer again.

Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "w…

The fundamental issue is that they are using a broken pattern for granularity of work. Any idiot can put together a lot of containers and farm work out, that's how Apache/CGI worked for years. But it's not scalable. They didn't understand the language runtime well enough, so they punted the problem to parallelism. At scale, they are now repairing the model to "save money" on what was essentially a no-architecture solution.
Post reply on HN