Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

221–230 of 261 posts

Re: We 30x'd our Node parallelism

#221

Post by GordonS »

rockostrich wrote: post #196
GordonS wrote: post #184

Earlier quoted context omitted.

Thanks for a reasoned response to what I realise was a very negative comment. I do agree with what you've said, and I do feel a little bad for slamming them when they're being transparent. OTOH, I do still feel this is so bad they need to be called out on it, and it really does scare me off using them. Given they're being transparent, it boggles the mind that they're tried to justify this, rather than just owning it,…

I posted it on my engineering org's random channel. The 4,000 instances of the same service thing immediately got a laugh out of everyone. How a tech company operated like this is beyond me...

Hah, I actually did the same, and it was basically a stream of WTFs?!

To put a positive spin on this (perhaps a first for me in this thread!), I plan on doing something of an internal post-mortem with my team, where we'll look at the deficiencies of this design, try to reason about how on earth it came to fruition, and critique our in-place review processes to make sure something like this never happens to us.

Re: We 30x'd our Node parallelism

#222

Post by z3t4 »

RSZC wrote: post #158
JMTQp8lwXL wrote: post #146

Earlier quoted context omitted.

Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.

Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.

One could argue that all errors are type errors. There is not just a type system, static or dynamic, inferred or annotated, which is good enough to catch all bugs. :P

Re: We 30x'd our Node parallelism

#223

Post by crazygringo »

bjacokes wrote: post #210
crazygringo wrote: post #99

Earlier quoted context omitted.

Yeah, I don't get it either, at all. The original poster wrote below: > In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc. But it's trivial (a single line) in Node to place breaks in CPU processing to allow the event loop to fire, and as for…

In one case I cited elsewhere in the comments, an engineer had called ramda.uniq on an array of nested objects which was occasionally very large. When calling into external packages, I don't think we have as much control over yielding to the event loop, but I could be wrong. I know that there are some JSON/regex libraries that give you some protection on this front. I agree that it would be nice if all developers wer…

Oh, interesting about external libraries.

Yeah, external libraries for Node ought to be designed so that any function that ever might take any length of time whatsoever should always be callable as async. But if they're badly designed or not intended for large inputs, they might not be. You'd definitely need to find another library or write your own there, so I get that.

Re: We 30x'd our Node parallelism

#224

Post by ubu7737 »

GordonS wrote: post #181
sicromoft wrote: post #162

Earlier quoted context omitted.

This comment says more about you than it does about Plaid. Their "insane" design met business requirements successfully enough to grow them into a multi-billion dollar company. Did you consider the likely (and more charitable) explanation that they were aware their design was "bad", but had higher priorities until now? If I were you, I'd be pulling your comment before it harms your reputation any further. :)

> Did you consider the likely (and more charitable) explanation that they were aware their design was "bad" I think marketing and VC valuations grew them into a multi-billion dollar company; whether they remain so, to a large part relies on how fast they burn through VC cash - so, not looking too good on that front... No even half-way competent engineer would come up with such a complex, unperformant solution to a si…

Downvoted for angering the VC class...

Re: We 30x'd our Node parallelism

#225

Post by ubu7737 »

throwGuardian wrote: post #174
GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

> no way I'm going near Plaid with a very long bargepole after reading this But you'd go to a competitor who hasn't published a blog post, whose internal code you haven't audited and simply presume is just fine? In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus.

ROFL "performance tuning" this is not tuning this is architecture.

Some people think you can just write software, sell it to customers, and it's "tuning" to make it work properly.

You should be fired from whatever job you have.

My guess is that you have no job, you are fronting USD.

In which case you have absolutely no place in this conversation and you should be ashamed of yourself for speaking up.

A fool and his money are easily parted.

Re: We 30x'd our Node parallelism

#226

Post by JMTQp8lwXL »

RSZC wrote: post #158
JMTQp8lwXL wrote: post #146

Earlier quoted context omitted.

Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.

Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.

A follow up, but I also wanted to highlight: plenty of typed systems (like Java Microservices) have bugs. If 100% of bugs were due to typing issues, these systems would -never- have bugs. Ever. Yet, we know that's not the case. That's another rationale for how the 1/4 to 1/3 less bugs ratio makes logical sense.

Re: We 30x'd our Node parallelism

#227

Post by acrispino »

GordonS wrote: post #219
bjacokes wrote: post #216

Earlier quoted context omitted.

I don't disagree with most of what you're saying. The Nth engineer at a startup rarely looks with admiration at decisions made by the (N/10)th engineer – but it was those decisions which helped the company grow to its current size. Likewise, I think most of us will be happy if the company 10x's again. Then some super-duper-senior engineer can look at the decisions we're making now – they're not perfect, but we're doi…

Gah, I can't believe you're still trying to justify this madness! In a parallel universe, a barely-competent engineer would have designed something more far more obvious, simpler, performant - all while using less hours , and not borderline-fraudulently wasting substantial amounts of your VC's money. If the company 10x's again, it won't be because of poor engineering, it'll be because of marketing and VC's who don't…

So, in your estimation, plaid's engineering is a bunch of less-than-half-competent madmen, who might as well be committing fraud, correct?

Is that overly negative or just the right amount?

Re: We 30x'd our Node parallelism

#228

Post by dang »

ubu7737 wrote: post #225
throwGuardian wrote: post #174

Earlier quoted context omitted.

> no way I'm going near Plaid with a very long bargepole after reading this But you'd go to a competitor who hasn't published a blog post, whose internal code you haven't audited and simply presume is just fine? In plaid's defense, lack of performance tuning isn't necessarily a lack of security focus.

ROFL "performance tuning" this is not tuning this is architecture. Some people think you can just write software, sell it to customers, and it's "tuning" to make it work properly. You should be fired from whatever job you have. My guess is that you have no job, you are fronting USD. In which case you have absolutely no place in this conversation and you should be ashamed of yourself for speaking up. A fool and his mo…

We've banned this account for breaking the site guidelines and ignoring our request to stop.

If you don't want to be banned, you're welcome to email hn@ycombinator.com and give us reason to believe that you'll follow the rules in the future.

https://news.ycombinator.com/newsguidelines.html

Re: We 30x'd our Node parallelism

#229

Post by dang »

PixyMisa wrote: post #82

Nobody involved in this project should be allowed to ever be in the same room as a computer again.

This comment breaks the site guidelines and is not cool. Would you please read https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here?

https://news.ycombinator.com/newsguidelines.html

Re: We 30x'd our Node parallelism

#230

Post by z3t4 »

GordonS wrote: post #149

I don't like to be overly negative, especially when a company/team is being transparent about what they're doing and giving insight into their engineering practices - but has anyone else's estimation of Plaid's engineering team just gone down the toilet? This blog post gives me the impression that either Plaid is filled with either junior or incompetent engineers - to scale to 4k containers serving 1 request each for…

Linus Torvalds quote: "You need to grow thick skin". I also got a knee-jerk reaction when reading the first part, but the article explained it well, given that they probably don't want to give out too much information.

So how would you have engineered it? I would just send the data uncompressed granted that the receiving server is probably in the same data-center with switches capable of handling Tbit's of data per second.

I liked the article, but would have wanted more details. I love optimizations, it's such a drug, the rush when you make something x times faster. This article doesn't give me a bad impression. Contrary I'm thinking about sending an application.

Post reply on HN