Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Node.js in Flame Graphs

techblog.netflix.com

Post reply on HN
Print view View original

141–150 of 259 posts

Re: Node.js in Flame Graphs

#141

Post by mikeryan »

jdmichal wrote: post #46
TheLoneWolfling wrote: post #13

> benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time 1ms / entry? What is it doing that it's spending 3 million cycles on a single path check?

Running (uncompiled?) regular expressions, it seems.

So I was a bit unclear on the parent's post but I don't think this time was on a route lookup if I'm reading the thread and post correctly the static file handler getting inserted multiple times. This handler will generally match on any route but then is doing something like "if file exists, return static file, if not look for the next handler" in this case the "if file exists" part was the "path check" thats taking 1 ms and was happening multiple times.

I could be wrong but it seems like the design of the route lookup mechanism (the global array) was actually a bit of a red herring, the real issue was the ability to attach multiple instances of the same handler to the same route.

Something was adding the same Express.js provided static route handler 10 times an hour. Further benchmarking revealed merely iterating through each of these handler instances cost about 1 ms of CPU time.

Re: Node.js in Flame Graphs

#143

Post by general_failure »

laughinghan wrote: post #114
general_failure wrote: post #49

Earlier quoted context omitted.

Netflix overhires quite a bit. They also pay their (very good) engineers very high salaries (possibly highest in the valley) which results is very low attrition. The net result is a lot of engineers having a lot of time and all these engineers experiment away on technology. In fact, they have quite a bit of NIH syndrome internally because the engineers have nothing to do.

I know I'm being a cocky asshole, but, are their engineers that good? > It’s unclear why Express.js chose not to use a constant time data structure like a map to store its handlers. Well, it was blindingly clear to me and the top commenter in this thread. I'm not judgmental that it was unclear to the author of this article, after all, I'm unclear on things that should've been stupidly obvious to me all the time. I am…

Well, the same engineer who didn't figure that out went ahead and profiled things, drew some pretty graphs, actually bothered to write a blog post to put his thoughts for public review. IMO, that is all some good engineering :-)

But you made me think what I said. I would say this sort of gets into the programmer / engineer debate. Netflix has good progammers like most companies. Good engineers (in my books) are very hard to find. These people are good at being at peace with tradeoffs and understand the requirements of a business at a much deeper level than programmers. Programmers just want to have fun with computers (and there is nothing wrong about that).

Re: Node.js in Flame Graphs

#144

Post by rgawdzik »

ChuckMcM wrote: post #130

The moneyquote: "We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue." This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varie…

You can't misuse closed source APIs? How would you know something is O(n) without seeing the source code?

Re: Node.js in Flame Graphs

#145

Post by degobah »

tl;dr:

* Netflix had a bug in their code.

* But Express.js should throw an error when multiple route handlers are given identical paths.

* Also, Express.js should use a different data structure to store route handlers. EDIT: HN commentors disagree.

* node.js CPU Flame Graphs (http://www.brendangregg.com/blog/2014-09-17/node-flame-graph...) are awesome!

Re: Node.js in Flame Graphs

#146

Post by baudehlo »

simi_ wrote: post #125
baudehlo wrote: post #80

Earlier quoted context omitted.

V8 compiles regular expressions to machine code.

You're the author of Haraka, a tool I'm prepared to try in production. This makes me _really_ worried that I'm making the right choice. (To explain: obviously C++ churns out machine code, the parent was talking about compiled vs uncompiled regexes – if I'm not terribly wrong, the compilation step is turning the regex into a finite automaton.)

V8 literally compiles regexps to X86 machine code the first time they are executed. They are not compiled into an FSA that gets walked in the traditional sense.

Hopefully that lowers your concern level.

Re: Node.js in Flame Graphs

#147

Post by gaving »

ecaron wrote: post #12

My biggest takeaway from this article is that Netflix is moving from Express to Restify, and I look forward to watching the massive uptick this has on https://github.com/mcavage/node-restify/graphs/contributors

not exactly highly endorsed https://twitter.com/tjholowaychuk/status/382084838179614721

Re: Node.js in Flame Graphs

#148

Post by vorador »

ChuckMcM wrote: post #130

The moneyquote: "We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue." This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varie…

> What this means in practice is that companies that use open source extensively in their operation, become slower and slower to innovate as they are carrying the weight of a thousand different systems of checks on code quality and robustness, which people using closed source will start delivering faster and faster as they effectively partition the review/quality question to the person selling them the software and they focus on their product innovation.

To contrast with what you said, I've worked at Microsoft, which is almost the company that invented NIH and we had the same problem. I think it's because, to paraphrase Alan Perlis, programming nowadays is less about building pyramids than fitting fluctuating myriads of simpler organisms into place.

Re: Node.js in Flame Graphs

#149

Post by dantiberian »

ChuckMcM wrote: post #130

The moneyquote: "We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue." This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varie…

Consequently any decision to use such an API seems to require one first read and review the entire implementation of the API, otherwise you get the experience that NetFlix had.

There's no getting around this, you or someone that you trust (not necessarily at your company) needs to read and review both the API, and implementation details for open source software that you use. Open source software isn't a hardware store you can dip into to get the latest parts that you need for your project. Adding a dependency makes your code depend on other peoples code. Just like internal code should go through a code review, so should external dependencies. This also explains why for a lot of companies, it's easier to write their own thing rather than need to keep on top of other people's changes.

I happened to write about this yesterday which explains my thoughts further http://danielcompton.net/2014/11/19/dependencies.

Re: Node.js in Flame Graphs

#150

Post by dpweb »

ChuckMcM wrote: post #130

The moneyquote: "We made incorrect assumptions about the Express.js API without digging further into its code base. As a result, our misuse of the Express.js API was the ultimate root cause of our performance issue." This situation is my biggest challenge with software these days. The advice to "just use FooMumbleAPI!" is rampant and yet the quality of the implemented APIs and the amount of review they have had varie…

Especially in high availability/load systems like Netflix IMO you need to reduce complexity and that means less modules that depend on 69 other modules, each of which depends on 69 modules etc.. What's going on as this sw proliferates is insane.

Then, you've got to be intimately familiar with every piece. There's no excuse when the source is readily available. I give these guys credit though they seem to take responsibility instead of just saying "express sucks". Some of their design choices seemed a little shaky.

Post reply on HN