Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We 30x'd our Node parallelism

blog.plaid.com

Post reply on HN
Print view View original

231–240 of 261 posts

Re: We 30x'd our Node parallelism

#231

Post by bjacokes »

GordonS wrote: post #220
bjacokes wrote: post #216

Earlier quoted context omitted.

I don't disagree with most of what you're saying. The Nth engineer at a startup rarely looks with admiration at decisions made by the (N/10)th engineer – but it was those decisions which helped the company grow to its current size. Likewise, I think most of us will be happy if the company 10x's again. Then some super-duper-senior engineer can look at the decisions we're making now – they're not perfect, but we're doi…

Ah, a serious question as an aside to my last (scathing) comment - does Plaid have architects? What about architecture and/or code reviews? I'd be very interested in reading about that.

Your question feels pretty sarcastic, but I think it'd be great to write a post about our process for projects specs and reviews. We do have an old post on the blog about code reviews that you could read, but it's more focused on the cultural approach to making code reviews less intimidating for junior engineers, less about the details. While we don't have an "architect" title, a fair portion of our more recent hires have 10+ years of experience (I don't have exact numbers though), and we're working on the best way to coordinate efforts and improve our system-wide architecture.

In general, I think it'd be great if more companies communicated their engineering practices and war stories externally – I'd love to read such posts myself! Unfortunately, it takes quite a bit of time to write a post (for engineers who are already stretched thin), and it seems that being honest about shortcomings at an early-stage company is an invitation for people to be personally disrespectful. It is what it is, but I imagine that's one reason we see such posts from only a small handful of startups, and the subject matter is often cherrypicked and sugarcoated.

So anyway, thanks for engaging with our post all the same – and maybe we'll be back with a post on architecture reviews in 5 years when all the kinks are ironed out :)

Re: We 30x'd our Node parallelism

#232

Post by z3t4 »

kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

Node.js is a good abstraction layer. In my experience, everything get leaky once you get hundreds of concurrent users.

Re: We 30x'd our Node parallelism

#233

Post by Wowfunhappy »

ubu7737 wrote: post #214
dang wrote: post #197

Earlier quoted context omitted.

Your comments in this thread have been breaking the site guidelines, and getting worse as they go along. Would you please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here? Note the bit about curious conversation .

Not clear. Edit: You want to be a moderator? Apply herein: https://www.newyorker.com/news/letter-from-silicon-valley/th... Just click on the big photograph and you'll be directed to the place where you can shut down conversation. Idiot. If you're curious then you should read and think.

I'm not sure if you're aware, but dang is one of the Hacker News moderators in that article. Dang stands for DANiel Gackle.

Re: We 30x'd our Node parallelism

#234

Post by jwalton »

kevstev wrote: post #104
x86_64Ubuntu wrote: post #96

Earlier quoted context omitted.

What does happen with the 5th request.

It will get queued, until one of the 4 requests in front of it has its task, to return the file, complete.

Well... yes. If you're using `express.static()`, or if all five requests are getting different things from the disk. If you're using something that caches your static content in-memory, then the first request will use the thread pool to read that content, but the other four requests won't touch the thread pool - it's all async IO at that point, so it's all happening "concurrently" in a single thread, being multiplexed right here: https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae...

Re: We 30x'd our Node parallelism

#235

Post by rclayton »

kevstev wrote: post #152
Nextgrid wrote: post #143

Earlier quoted context omitted.

You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea. Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?

I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else. However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types"…

Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.

Re: We 30x'd our Node parallelism

#236

Post by bathtub365 »

JMTQp8lwXL wrote: post #167
RSZC wrote: post #158

Earlier quoted context omitted.

Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors. I'd love to see any data or case studies that claim the opposite if you have any.

https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q... > Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?... https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu... > 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis I've never seen a number far outside of the 15-30% range. In my experience, most bugs are…

How is JavaScript worth it if it typically adds 15-30% more bugs? That’s an enormous figure.

Re: We 30x'd our Node parallelism

#237

Post by rclayton »

kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

Is it really only 4? When I look at my VM stats I seem to recall it having something like 15. Of course, this could be a config change on the Node Alpine container.

Re: We 30x'd our Node parallelism

#238

Post by rclayton »

Kiro wrote: post #141
kevstev wrote: post #67

I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, t…

I have a Node service where I get tens of thousands requests a second and I still thought Node was single threaded. Where can I read about this?

The event loop is single threaded. Async tasks are executed on I/O threads which are configurable. So if your app is I/O bound, the event loop will typically dequeue tasks pretty quickly allowing lots of requests.

Re: We 30x'd our Node parallelism

#239

Post by JMTQp8lwXL »

bathtub365 wrote: post #236
JMTQp8lwXL wrote: post #167

Earlier quoted context omitted.

https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q... > Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?... https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu... > 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis I've never seen a number far outside of the 15-30% range. In my experience, most bugs are…

How is JavaScript worth it if it typically adds 15-30% more bugs? That’s an enormous figure.

Well, it runs everywhere, for one. That's an attractive quality to a language.

Re: We 30x'd our Node parallelism

#240

Post by mceachen »

contrahax wrote: post #133
mceachen wrote: post #38

In case anyone else gets excited by JSONStream, know that the package hasn't been updated in over a year, and the GitHub repo was archived by the author with no link to a successor.

I'm maintaining a fork here that incorporates all of the valid open PRs from the original repo + some more updates: https://github.com/contra/JSONStream It isn't published on NPM (you can use it as a git dependency) but if people are interested I can.

Thanks for sharing!

Why don't you publish releases?

Post reply on HN