We 30x'd our Node parallelism • Hacker News Board

Re: We 30x'd our Node parallelism

#191

Post by tracker1 » Fri, Dec 13, 2019, 9:14 PM UTC

praseodym wrote: post #83

7777fps wrote: post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

Related, from the article: > We hypothesized that increasing the Node maximum heap size from the default 1.7GB may help. To solve this problem, we started running Node with the max heap size set to 6GB [..], which was an arbitrary higher value that still fit within our EC2 instances. Sounds like they were utilizing their EC2 instances very poorly. Why not run more workers per instance, or switch to an instance type w…

They were using ECS, it also looks like they had to work through a couple bottlenecks to get multiple requests per node working well... I think they could get further by using the newer workers api, since gRPC doesn't work with the cluster module.

Re: We 30x'd our Node parallelism

#192

Post by phoe-krk » Fri, Dec 13, 2019, 9:18 PM UTC

megous wrote: post #122

phoe-krk wrote: post #60

Earlier quoted context omitted.

For these kinds of programming, yes, I argue against it. A single stalling function in Node deadlocks the whole system; a single stalling thread in the C++ model still permits other threads to run. This is a risk that is completely avoidable by not using languages which require event loops at their core.

JavaScript doesn't require event loop design. You can do a PHP like backend design with JS, where each request is handled by a fresh process, and all JS functions block. There's nothing in the language that prevents this. Some features would become unusable, like Promises and async/await, but those would be worthless in such a design anyway.

Nobody does it. Nobody implemented it yet. Nobody made it production ready. That is what a few minutes of googling gave me.

As much as I'd like to see it, your JS without an event loop is a purely theoretical construct so far.

Re: We 30x'd our Node parallelism

#193

Post by ubu7737 » Fri, Dec 13, 2019, 9:18 PM UTC

tracker1 wrote: post #177

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons incl…

Where I work compliance is job #1.

That doesn't prevent us from thinking about performance. GTFO with this nonsense.

Re: We 30x'd our Node parallelism

#194

Post by ubu7737 » Fri, Dec 13, 2019, 9:20 PM UTC

mharroun wrote: post #171

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

In their defense. It looks like they have over 400 employees and raised over 350 million in funding. On all things that truly matter currently they seem like a very sucessfull company. I can guarantee you a VPE or CTO who can say they helped do that... but ran into a scaling issue from their success will have no issue with employment and no reason to be ashamed. All the more impressive if it was just a bunch of junio…

"In their defense: money they've raised"?

Is this supposed to be a contribution to the thread?

Re: We 30x'd our Node parallelism

#195

Post by tracker1 » Fri, Dec 13, 2019, 9:20 PM UTC

neillyons wrote: post #87

MayeulC wrote: post #72

Earlier quoted context omitted.

> [javascript] makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies. As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red b…

In Python, you can execute time.sleep(10_000) in an asyncio program, which means it wont yield back control to the event loop effecting preventing the runtime from doing anything for 10 seconds. In Javascript world I guess you could do the same by replacing time.sleep with some CPU bound code. eg a big calculation or an infinite for loop.

Yes... often JSON serialization and deserialization are the biggest blockers, if they're using puppeteer (as suggested in other comments) that can have its' own instance overhead. I think with ECS they used the "easy" button to start off with and now that they've hit those limits have had to refactor, which isn't unreasonable imho.

I'm working on a project now that's very CPU bound and using limited workers behind an MQ as a distributed RPC behind a fronting API interface so that I can handle scaling... though it's also a Windows-only library involved. There's definitely an art to scaling certain types of workloads and many different options. Sometimes the simplest solution you can come up with really is the best option.

Re: We 30x'd our Node parallelism

#196

Post by rockostrich » Fri, Dec 13, 2019, 9:21 PM UTC

GordonS wrote: post #184

yjftsjthsd-h wrote: post #156

Earlier quoted context omitted.

I mean, that's always the thing, isn't it? If a company publishes about the problems it has, the question is whether other companies have the same problems and just hide it, or whether this company is actually worse. This comes up a lot with gitlab, for instance; remember the time they discovered they had no backups? At most companies, customers would never find out about that, so I'm not sure that them telling us ab…

Thanks for a reasoned response to what I realise was a very negative comment. I do agree with what you've said, and I do feel a little bad for slamming them when they're being transparent. OTOH, I do still feel this is so bad they need to be called out on it, and it really does scare me off using them. Given they're being transparent, it boggles the mind that they're tried to justify this, rather than just owning it,…

I posted it on my engineering org's random channel. The 4,000 instances of the same service thing immediately got a laugh out of everyone. How a tech company operated like this is beyond me...

Re: We 30x'd our Node parallelism

#197

Post by dang » Fri, Dec 13, 2019, 9:21 PM UTC

ubu7737 wrote: post #193

tracker1 wrote: post #177

Earlier quoted context omitted.

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons incl…

Where I work compliance is job #1. That doesn't prevent us from thinking about performance. GTFO with this nonsense.

Your comments in this thread have been breaking the site guidelines, and getting worse as they go along. Would you please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here? Note the bit about curious conversation.

Re: We 30x'd our Node parallelism

#198

Post by ubu7737 » Fri, Dec 13, 2019, 9:22 PM UTC

sicromoft wrote: post #162

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company. Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now? If I were you, I'd be pulling your comment before it harms your reputation any further. :)

[dead]

Re: We 30x'd our Node parallelism

#199

Post by tracker1 » Fri, Dec 13, 2019, 9:22 PM UTC

x86_64Ubuntu wrote: post #92

jononor wrote: post #40

Earlier quoted context omitted.

The blog in question indicates some uncommon and questionable engineering practices with Node.js. There are likely hundreds of success stories for every one like that. The first Node.js service I wrote and maintained, processed thousands of requests in parallel and was successfully in production until the company it was developed for ran out of money.

But how many of these "questionable" practices get held up as engineering marvels by the creators of the monstrosity? When you look at this blog post, you can see the author really felt a sense of pride in this Frankenstein and wanted to show the whole village.

I didn't get that impression.. what I got was a few probably common issues that will come up when trying to handle more parallel requests with Node. Note: gRPC doesn't support cluster, so you'd have to manage your own multi-thread (Workers) or multi-process (fork) options.

Re: We 30x'd our Node parallelism

#200

Post by rockostrich » Fri, Dec 13, 2019, 9:27 PM UTC

RSZC wrote: post #155

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

It's important to keep in mind that efficiency isn't usually particularly important for a startup. I'm sure they knew when they initially set up this system that it wasn't performant...but it was nice and quick and easy and gets the feature out the door. Why should they worry about $100k or whatever when they're funded for > $350M? Their bottleneck is engineer hours, not dollars. Instead the rational thing to do is b…

>Their bottleneck is engineer hours, not dollars.

Their bottleneck was software being able to scale past a hard stop. I guess having a known breaking point of scalability is a good thing? But building things in a way where you either have to overhaul your development runtime or not be able to scale past a certain point is pretty terrible.

It seems like the only reason they did this was because they really felt the pain of it from the business and dev side and they were lucky enough that they had traffic spikes to raise these issues. If they had more consistent day-to-day traffic then this would have just hit a breaking point one day and they would've been fucked until it was fixed.