Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node.js in Flame Graphs

techblog.netflix.com

Post reply on HN
Print view View original

51–60 of 259 posts

Re: Node.js in Flame Graphs

#51

Post by yourad_io »

yourad_io wrote: post #39
remon wrote: post #21

I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.

What are the arguments against node.js in their use case? Not looking to start any wars, but I was under the impression that if you know what you're doing* node.js is pretty awesome. This particular bug had to do with a misunderstanding regarding the express API. * for the most part: understand async and closures/memory leaks.

To give some arguments in favour:

- "Here we see our latencies drop down to 1 ms and remain there after we deployed our fix." - They can hire from a vastly larger pool of developers. Since they've got both budget and brand recognition, they should have no problems hiring high end JS guys & gals.

Re: Node.js in Flame Graphs

#52

Post by guiomie »

general_failure wrote: post #49
personZ wrote: post #33

Earlier quoted context omitted.

Netflix seems to operate like a tech start-up that is trying to glue together a ragtag collections of often unsuitable solutions because of limited funding. It is a deeply perplexing company. Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regul…

Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

NIH ?

Re: Node.js in Flame Graphs

#53

Post by Pharohbot »

I wonder how Netflix would perform with using Dart with the DartVM. I reckon it would be faster than Node based on benchmarks I've seen. Chrome DartVM support is right around the corner ;)

Re: Node.js in Flame Graphs

#54

Post by mjs »

andrewvc wrote: post #36
thedufer wrote: post #10

> It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers. Its actually quite clear - most routes are defined by a regex rather than a string, so there is no built-in structure (if there's a way at all) to do O(1) lookups in the routing table. A router that only allowed string route definitions would be faster but far less useful. I can't explain away the recursion…

A lot of people here are right, the right way is with an NFA. I just want to add that the solution is not even hard, you can do it with string concatenation and capture groups using regexps. Regexps are NFAs, and are highly optimized C code in just about every JS engine. If I have the routes /foo/bar and /foo/bar/(\d+) I can generate the regexp ((^\/foo\/bar$)|(^\/foo\/bar\/\d+$)) I'm not at all surprised, the qualit…

It looks like routes are either strings (exact match) or regular expression objects. I don't think there's a way to combine regular expressions in JavaScript.

I don't think this was an unreasonable implementation. Sure, it could be optimized, but no-one needed it enough to do the work. (Even Netflix didn't need a faster matcher, they needed a matcher that didn't slow down between the 1st match and the nth.)

Re: Node.js in Flame Graphs

#55

Post by general_failure »

guiomie wrote: post #52
general_failure wrote: post #49

Earlier quoted context omitted.

Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

NIH ?

Not invented here. Redoing stuff which already exists.

Re: Node.js in Flame Graphs

#56

Post by quotemstr »

thedufer wrote: post #35
colanderman wrote: post #31

Earlier quoted context omitted.

No. n in the case of a DFA is the length of the input string. (i.e. the string being matched.) So, in terms of the GP's post, yes, this is an O(1) (with respect to number of routes) solution.

Oh, right. My CS has gotten fuzzy - the tree gets enormous, but the runtime stays is O(n) on input length. Thanks for the explanation.

The automaton isn't even that big, really. The number of NFA states is roughly proportional to the total number of characters in the regular expressions involved, and NFA to DFA conversion usually expands the automaton only be a factor of two or three. Although the subset construction in theory worst-case exponential, that situation never occurs in practice.

Re: Node.js in Flame Graphs

#57

Post by ecnahc515 »

sadkingbilly wrote: post #42
ecaron wrote: post #12

My biggest takeaway from this article is that Netflix is moving from Express to Restify, and I look forward to watching the massive uptick this has on https://github.com/mcavage/node-restify/graphs/contributors

Yes, but their original bug was from dynamically loading routes from an external source. I don't see how Express is to blame for this. Moving to Restify is not a solution, but they state having different reasons for moving (support for bunyan logging? But Express already supports this too).

the bug was related to dynamically loading routes, but the true cause was that express allowed duplicate handlers. They were loading routes dynamically correctly, that wasn't a problem, it was that when doing that express let them duplicate routes.

Re: Node.js in Flame Graphs

#58

Post by codelucas »

> This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array. Unfortunately, it was also inadvertently adding a static route handler with the same path each time it ran.

I don't understand the need of refreshing route handlers. Could someone explain they needed to do this, and also why from an external source?

Re: Node.js in Flame Graphs

#59

Post by eikenberry »

general_failure wrote: post #49
personZ wrote: post #33

Earlier quoted context omitted.

Netflix seems to operate like a tech start-up that is trying to glue together a ragtag collections of often unsuitable solutions because of limited funding. It is a deeply perplexing company. Similar is LinkedIn, as an aside -- despite being fairly formidable now, I regularly have entire feeds disappear, their caching is abhorrent, they can't markup text properly, and so on. It seems very amateur hour, yet they regul…

Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

I don't think you have those reasons in proper order. I think the ability of the engineers to experiment and use new tools is why they have the high level of retention. I've seen plenty of places that pay much higher than the competition yet have very high turnover due to the a conservative culture which dictates the tools and doesn't encourage experimentation.

Re: Node.js in Flame Graphs

#60

Post by pcl »

Second, given a performance problem, observability is of the utmost importance

I couldn't agree with this more. Understanding where time is being spent and where pools etc. are being consumed is critical in these sorts of exercises.

Post reply on HN