Hard-won lessons: Five years with Node.js • Hacker News Board

Re: Hard-won lessons: Five years with Node.js

#361

Post by thibaut_barrere » Sun, Apr 23, 2017, 7:21 PM UTC

jononor wrote: post #198

thibaut_barrere wrote: post #3

Something I found surprising was the 1.5 GB limit per process when used on Heroku ( https://devcenter.heroku.com/articles/node-concurrency ), which afaik can be circumvented elsewhere ( https://futurestud.io/tutorials/node-js-increase-the-memory-... ). This can be problematic at times if you need to open a very large number of concurrent connections (crawlers, slack connections etc), and can force you to organize a "…

You can have quite a number of concurrent connections in 1GB of RAM, 10k at least? That seems allright for a single node, and scaling horizontally after that. For something like crawling, which has a lot of time spent on CPU intensive parsing and I with unpredictable times (hitting external service) I would throw that in a background worker from the start.

I do not have the numbers handy, but it was below 10k (by quite a margin), unfortunately.

Re: Hard-won lessons: Five years with Node.js

#362

Post by jklepatch » Tue, Apr 25, 2017, 2:35 AM UTC

sgift wrote: post #142

eel wrote: post #67

Earlier quoted context omitted.

At a previous gig, without getting into specifics, we built a huge internal client-server system. The client was written in Node.js and ran as services on both Windows and Linux on a few thousand machines. The server was written in Node.js and ran as a Linux daemon. Node.js matched the developers' skill sets and gave us the ability to quickly write a server that could easily handle several thousand concurrent request…

> * Target environment is not Windows That one is still my main problem. I know, I know, Windows is evil and so on, but our customers use it. We are remarkably free in choosing our tools or whatever, but Windows is usually a constant (Intranet environments) and every day I crash against the absolute "Windows is a second class platform" problem of NPM modules. For an environment which is supposedly platform-agnostic t…

Really? I think exactly the opposite. Compared to a lot of other tech, I found nodejs to work pretty well on windows. For example, recently npm switched from a nested to a flat directory organisation in node_modules. Windows has a limit on the number of nested directories, so this new organisation helped a lot.

Maybe that you use a lot of modules with c dependencies?

Re: Hard-won lessons: Five years with Node.js

#363

Post by dahauns » Tue, Apr 25, 2017, 10:00 AM UTC

z3t4 wrote: post #228

dahauns wrote: post #209

Earlier quoted context omitted.

I have a hard time really seeing this as a net win. There are huge upsides to having the same language on both sides - but the big downside is that javascript simply isn't a very good backend language . Node.js/v8 is incredibly impressive for what it is. But it still has "shoehorned" written all over it. (For the record: I wouldn't consider Java the Language a pinnacle either.)

I agree that JavaScript isn't very good, but what are the alternatives !? I can not think of any language where a non-engineer with little computer science knowledge, can get productive withing minutes. And it only gets better the more you use it. For example, the other day I wanted to take a screen-shot of a web page, make a visual diff, then email me if something changed with the changes highlighted. I basically gl…

> I can not think of any language where a non-engineer with little computer science knowledge, can get productive withing minutes.

That's a big misconception there. I absolutely don't mean this in an elitist way - and I completely agree that some degree of accessibility is a must - but this is in no way a benchmark for a good backend language. This is how we got PHP.

> [WebPage screenshot -> visual Diff -> send mail] > Try to do that in any other language.

I fail to see what's so difficult about this. I'd wager that in the "heavyweight stacks" you'd not even have to use modules outside the runtime libraries for a basic variant, and if you want to, you'd have them at your disposal.

Don't get me wrong, the large centralized ecosystem of NodeJS definitely is one of its biggest pluses.

But - and yeah, now I'm sounding a bit elitist and jaded - I've heard a variant of "It's so much simpler in X" or "Try that in anything else than X" too often when it only meant "I'm actually not really proficient in anything else but X".

Re: Hard-won lessons: Five years with Node.js

#364

Post by [deleted] » Wed, Apr 26, 2017, 2:24 PM UTC

chrisco255 wrote: post #165

lod723 wrote: post #144

Earlier quoted context omitted.

The multiprocess Node concurrency model is brittle and fault intolerant. The event loop scan actually has real bottlenecks at a certain level of fds in flight. A real scheduler really wins here; libuv sits below the knowledge of the runtime to really be optimal. You become CPU bound far sooner than you expect. With a global heap, you also become memory bound far sooner than you expect as well.

Your typical Node instance is using an order of magnitude less memory than the beastly JVM. We use Java microservices with Spring at my job and each "micro" service consumes close to 1GB of memory. I've never written a Node service that had more than 100MB footprint.

[deleted]

Re: Hard-won lessons: Five years with Node.js

#365

Post by lod723 » Wed, Apr 26, 2017, 2:57 PM UTC

chrisco255 wrote: post #165

lod723 wrote: post #144

Earlier quoted context omitted.

The multiprocess Node concurrency model is brittle and fault intolerant. The event loop scan actually has real bottlenecks at a certain level of fds in flight. A real scheduler really wins here; libuv sits below the knowledge of the runtime to really be optimal. You become CPU bound far sooner than you expect. With a global heap, you also become memory bound far sooner than you expect as well.

Your typical Node instance is using an order of magnitude less memory than the beastly JVM. We use Java microservices with Spring at my job and each "micro" service consumes close to 1GB of memory. I've never written a Node service that had more than 100MB footprint.

The JVM has the same global heap problem as Node... it's frustrating that with all the development that goes into the JVM this problem remains unaddressed.