Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

201–210 of 261 posts

Re: We 30x'd our Node parallelism

#201

Post by tracker1 »

GordonS wrote: post #185
tracker1 wrote: post #177

Earlier quoted context omitted.

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons incl…

> For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. I get it, but come on - this was not a "performance optimisation" issue, but one of bad architecture; an architecture that certainly doesn't inspire confidence in the priorities you mention: accuracy…

Having a worker that does one request, processes that one request and returns a result isn't accurate, simple or safe? Scaling that simplistic interface in a 1:1 manner across many systems via ECS is pretty straight forward.

Now, in addition to probably optimizing what they've done, converting to, for example another container system, like K8s where they can scale vertically a bit better may have been another approach.

The biggest issue that I see is gRPC doesn't work that great with Node. You can't use it with cluster which means you have to self-manage threads/processes and it adds complexity there. Yeah, there's definitely issues that come into scaling in terms of performance optimization.... but where they started from isn't unreasonable imho.

Re: We 30x'd our Node parallelism

#202

Post by tracker1 »

ubu7737 wrote: post #193
tracker1 wrote: post #177

Earlier quoted context omitted.

For banking... accurate, simple, safe, reliable are more important than performance/throughput. IMHO optimizing the above and for developer efficiency should be the first priority and for scale or max throughput later. The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons incl…

Where I work compliance is job #1. That doesn't prevent us from thinking about performance. GTFO with this nonsense.

They started with a simple implementation where one node handles one request at a time... end to end. They used ECS for easy-button scaling. That's a perfectly reasonable approach for starting out.

I would have probably pushed for a shift in orchestration to kubernetes along with some tweaking as an initial uplift. Others would re-write the whole thing in another language. They chose to add a bit of complexity for multiple requests per node support. They all have their pluses and minuses, but in the end it doesn't mean the initial approach was bad, or their refactor wasn't pragmatic or practical.

Dramatic rewrites to a codebase lead to instability and in practice fail as much as succeed.

Re: We 30x'd our Node parallelism

#203

Post by rockostrich »

sicromoft wrote: post #162
GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company. Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now? If I were you, I'd be pulling your comment before it harms your reputation any further. :)

>multi-billion dollar company

WeWork is a "multi-billion dollar company" in the same way that Plaid is. Private funding valuations don't really mean anything anymore.

Re: We 30x'd our Node parallelism

#204

Post by bjacokes »

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

Hi, Plaid engineer here (not the author, but I helped with the post).

I don't think we've tried to assert that the old system is perfect. We went into some detail in the post about why it took us this far. Certainly, the single request per container approach wouldn't scale if our unit economics were different. We didn't get into this too much in the post, but the Node service sits behind a couple of layers of Go services, so the we had more control over scaling API traffic than it might appear.

Likewise, I hope we didn't give the impression that the new system is perfect. We've explored other languages for integrations in the past (even Haskell, at one point), and are continuing to do so. A migration away from our years-old Node integrations codebase would be a massive undertaking at this point. Absent that, it doesn't seem consistent to say "you're incompetent for handling 1 request per container" and also "you're incompetent for writing this post" – if you believe the former then it makes sense to be an advocate for this project, at least until a language migration can be done.

I think the set of hoops we had to jump through in order to add concurrent requests without adding latency is a good demonstration of why we didn't do this sooner. It wasn't a massive undertaking by any means, but it wasn't trivial. At any rate, we're not really looking for a gold star here – just putting this out there and hoping this will be useful for others who are, as other commenters have put it, building their own "Frankensteins" :)

Re: We 30x'd our Node parallelism

#205

Post by a13n »

Nextgrid wrote: post #143
a13n wrote: post #134

Earlier quoted context omitted.

> safety and correctness features You can achieve safety and correctness features for node via good lint rules and typescript/flow.

You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?

We chose JavaScript/Node because it's the language we knew best, and because we have to write it in the client as we're building a web-based service. No other language offers us that, unless it compiles to JavaScript (eg TypeScript).

This grants us a magnitude of benefits, like being able to do server-side rendering and share code between the client and server. It also means we only need to hire people who can write JS, instead of JS and something else.

TypeScript, Flow, and ESLint aren't hacks. These are mature tools used by some of the largest, most sophisticated engineering teams in the world.

Re: We 30x'd our Node parallelism

#206

Post by [deleted] »

golergka wrote: post #148
kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

> When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that. It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all? I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always…

[deleted]

Re: We 30x'd our Node parallelism

#207

Post by tyingq »

godot wrote: post #107
tyingq wrote: post #6

Does node have something similar to how apcu is used with PHP? That is, an mmap based kv store so that if you choose to run more than one node process on a single server, it has a fast kv cache? I'm aware you can use redis or similar, but a simple mmap kv store is simpler and faster for a single server use case.

I totally see what you mean, coming from a PHP world myself a few years ago. The key thing to note is that node.js (like many other languages including Java) starts a server process that basically does not stop until you explicitly restart it (or it crashes); unlike PHP where every request starts a brand new process on a clean slate (hence needing APCu to store a local memory cache per server). Meaning, what you can…

The context is multiple node processes running on a single box, so a shared cache across processes has value for some use cases. I don't think the cache module you suggested would work in that case.

I'm aware of the runtime model differences between node and PHP.

Re: We 30x'd our Node parallelism

#208

Post by [deleted] »

tracker1 wrote: post #202
ubu7737 wrote: post #193

Earlier quoted context omitted.

Where I work compliance is job #1. That doesn't prevent us from thinking about performance. GTFO with this nonsense.

They started with a simple implementation where one node handles one request at a time... end to end. They used ECS for easy-button scaling. That's a perfectly reasonable approach for starting out. I would have probably pushed for a shift in orchestration to kubernetes along with some tweaking as an initial uplift. Others would re-write the whole thing in another language. They chose to add a bit of complexity for mu…

[deleted]

Re: We 30x'd our Node parallelism

#209

Post by yurish »

sho wrote: post #154
spamizbad wrote: post #14

The only way this makes sense to me is if they have to contend with lots of expensive parsing, event sequencing, and throttling requirements. Payment APIs, bank websites, etc can be quite byzantine. I could understand how one might code yourself into a corner with a monolithic node app and basically just say "F-it, we're doing this synchronously!" I don't even think it's a terribly bad thing to do assuming it favors…

> explore Go or Elixir I have never seen a good argument for using golang for business logic. If you are writing the actual server then sure, use golang. If you are writing some high-speed network interconnect, use golang. Some crazy caching system, sure use golang. The public WS endpoint, use golang. But if you need to access a DB with golang for anything more than, like, a session token, then you made the wrong cho…

What is wrong with accessing DB from golang?

Re: We 30x'd our Node parallelism

#210

Post by bjacokes »

crazygringo wrote: post #99
7777fps wrote: post #2

> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop, and allowed us to ignore the variability in resource usage across different integrations. But since our total capacity was capped at 4,0…

Yeah, I don't get it either, at all. The original poster wrote below: > In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc. But it's trivial (a single line) in Node to place breaks in CPU processing to allow the event loop to fire, and as for…

In one case I cited elsewhere in the comments, an engineer had called ramda.uniq on an array of nested objects which was occasionally very large. When calling into external packages, I don't think we have as much control over yielding to the event loop, but I could be wrong. I know that there are some JSON/regex libraries that give you some protection on this front.

I agree that it would be nice if all developers were infallible – I'm reminded of a friend describing their company, where "we don't write tests because we all write good code". At a certain point, you have to look for processes – linters, monitoring, testing, language choices [1] – where people can't shoot themselves in the foot. (Code reviews being only moderately less fallible than a single engineer.) It's not enough to just say "be better" whenever bad code is written.

I think when the decision was made (years ago) to handle a single request per container, they couldn't find such a process to prevent event loop blockages, other than migrating an already-large codebase away from Node. As others have pointed out, maybe such a migration is necessary – after all, event loop blockages are still an inherent risk because of how Node works. It's just a lower risk than it was a year or two ago, because we've significantly improved our usage of the event loop, and also have tooling in place to catch blockages before they become an issue.

[1] https://news.ycombinator.com/item?id=18564643

Post reply on HN