Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

161–170 of 261 posts

Re: We 30x'd our Node parallelism

#161

Post by dwild »

PixyMisa wrote: post #116
jrockway wrote: post #111

Earlier quoted context omitted.

Why? They had a 12 factor -ish app that scaled the normal way; run more copies. Eventually that got expensive. They had the observability to figure out what was making it expensive and whether or not their fixes had an effect. They then saved $300,000. Seems like everything went right to me. I would be worried if the blog post was "we randomly tweaked some stuff and we can't measure it but it's a little better" or "w…

To save $300,000 they first needed to waste $300,000 by reinventing a problem that was solved in 1967.

I don't know how many software engineer they got on that team, but considering how much they raised and how much their product is used, $300k seems actually quite cheap for something that people consider here as being an awfully big mistake.

Re: We 30x'd our Node parallelism

#162

Post by sicromoft »

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company.

Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now?

If I were you, I'd be pulling your comment before it harms your reputation any further. :)

Re: We 30x'd our Node parallelism

#163

Post by kevstev »

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

I kind of alluded to it in my reply, but I tend to agree- they spent a lot of time and hard work- looking in all the wrong places! Its hard to imagine how they missed the forest for the trees so badly here.

Worse is- they never really explain where that 30x improvement came from- or if they even understand it themselves? They talk a lot about getting their memory issues under control, but hardly at all about actual parallelism- and it seems that even then they confuse it with merely speeding up operations that are blocking.

I kind of expected this post to be "We did a whoops and had a blocking call to a DB/fs/compression call/whatever. This was all happening in the event loop and not being farmed out to the threadpool by libuv. We fixed it and now look like heroes to our CTO!"

Re: We 30x'd our Node parallelism

#164

Post by Nextgrid »

bfrydl wrote: post #153
Nextgrid wrote: post #143

Earlier quoted context omitted.

You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?

I find it really annoying how JavaScript is treated on developer forums like this one. Why is it that when the same exact things are done in a typical language it's called “tooling” but in JavaScript it's “duct tape hacks on top of hacks”? Don't get me wrong, my favorite language is Rust, but pretending the JavaScript ecosystem is unusable doesn't make you cool. I can be extremely productive in TypeScript.

JavaScript itself is not a hack, it works fine for its original purpose - making webpages interactive.

What I consider hacks is the hundreds of different tools and dialects of JS that are used to bend JS into doing something it wasn’t really designed for.

JS is fine in the browser, and TypeScript is also fine there because you don’t really have the choice to run anything different (although Web Assembly might change the game soon). But on the backend you have the privilege to pick between dozens of different languages that are better suited to the task and support the features you want out of the box without layering hacks on top. Why not just go with one of those?

Re: We 30x'd our Node parallelism

#165

Post by ubu7737 »

markandrewj wrote: post #132
crazygringo wrote: post #129

Earlier quoted context omitted.

I get that. What I don't get is how nobody treats it as an issue when developers coming from Python or Java to C don't get pointers. The assumption is that you learn . But for some reason, people think it's "OK" to not get async, that it's the language's fault rather than the programmer's. That's what I don't understand. It's like a different cultural standard gets applied.

I agree with you, but I think many developers get reluctant to change when they have been doing something one way for a long time, especially if they feel that one way works fine. I can also understand the position, as sometimes it can be fatiguing when technologies are constantly changing. For this project though, if they are actively going to avoid asynchronous programming, they may have been better off choosing a…

There is simply no excuse for this. You are a software engineer or you are not, the cadence of change is part of the technology aspiration.

I have no patience for persons who don't belong to the discipline.

Re: We 30x'd our Node parallelism

#166

Post by kevstev »

RSZC wrote: post #158
JMTQp8lwXL wrote: post #146

Earlier quoted context omitted.

Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.

Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.

Just curious- did you use Joi or anything similar to try to at least verify at some point that you had a valid object?

Re: We 30x'd our Node parallelism

#167

Post by JMTQp8lwXL »

RSZC wrote: post #158
JMTQp8lwXL wrote: post #146

Earlier quoted context omitted.

Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.

Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.

https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q...

> Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?...

https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu...

> 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis

I've never seen a number far outside of the 15-30% range.

In my experience, most bugs are operator error. Developers didn't code for branching paths that should've been accounted for, etc.

Personally, I'm a fan of TypeScript. Just don't expect to remove the majority of your bugs via its usage. The old "no silver bullet" adage.

Re: We 30x'd our Node parallelism

#168

Post by stickfigure »

rdsubhas wrote: post #125
stickfigure wrote: post #74

Earlier quoted context omitted.

I'd recommend moving away from Node... Taking a wild guess: Some of their bank integrations probably require browser automation. If you're doing browser automation, the best tool for the job is (currently) Puppeteer, which runs on Node. There are other third-party language bindings for the Chrome dev tools protocol, but Puppeteer is developed by Google as a first-class citizen alongside Chrome.

4000 chrome instances? Probably not. Here I am trying to run 4 chrome instances in parallel in CI without crashing.

Presumably not every integration requires browser automation, so they might not all be going at once. But they have a $25k monthly EC2 bill, so it's not out of the ballpark.

FWIW, I reliably have 6 puppeteer/chrome instances (headful, even) going on a single box and it's not even at half capacity.

Re: We 30x'd our Node parallelism

#169

Post by ubu7737 »

dwild wrote: post #161
PixyMisa wrote: post #116

Earlier quoted context omitted.

To save $300,000 they first needed to waste $300,000 by reinventing a problem that was solved in 1967.

I don't know how many software engineer they got on that team, but considering how much they raised and how much their product is used, $300k seems actually quite cheap for something that people consider here as being an awfully big mistake.

Unfortunately it never works out the way you hope it will, once the VC money shows up. ROI will become even more of a pattern for everything, which means you burn the existing platform to dust.

Or you hire a team of actual engineers to turn the PoC into an actual product.

My judgment tells me the management at Plaid doesn't know how to tell the difference, so it's sort of a determined outcome.

Re: We 30x'd our Node parallelism

#170

Post by bjacokes »

ubu7737 wrote: post #139
bjacokes wrote: post #131

Earlier quoted context omitted.

While we were worried about event loop blockages causing outages, another more subtle problem would have been if event loop blockages doubled our user-facing latency. (If you read the section on latency ratios, you'll see that comparing parallel vs non-parallel workers was the most useful stat in figuring out how effectively we were using the event loop.) It definitely gave us peace-of-mind to know that event loop bl…

> But it's possible this migration process would have been easier if we had done it sooner. What the f *? Of course it would have been easier if you had done it sooner. What you lacked was the willpower from decision-makers who had growth of dollar-signs in their eyes. You've littered this thread with comments explaining how every move you made was based on ROI. That's the kiss of death for architecture concerns, and…

I say "possible" because our system observability was less mature even 12 months ago. Firefighting 10 different root causes of memory or event loop issues without the right tooling in place would be a nightmare. That's why we did a deep dive into the tooling that we considered to be a prerequisite for this project – hopefully it's helpful for others in our situation.

Different companies make different decisions when weighing ROI against architecture concerns. We're heavy on pragmatism and impact at Plaid, so it's quite intentional that we don't fall all the way on the latter end of the spectrum. I appreciate the discussion in the comments as to how effectively we are balancing these two concerns – certainly this is an area where reasonable people can disagree.

Post reply on HN