I am upset that the title has been changed from "Node.js in Flames". Which is not only the real title of the article, but also a reasonable description of what they've been facing with Node.
#moderationfail
Node.js in Flame Graphs
221–230 of 259 posts
Re: Node.js in Flame Graphs
#222I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.
I share this thought, I'm not trolling, I really believe node is a bad solution for something like Netflix. Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much. I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opi…
Could you expand more on the ops issues you have encountered in your systems?
Re: Node.js in Flame Graphs
#223Earlier quoted context omitted.
Running (uncompiled?) regular expressions, it seems.
So I was a bit unclear on the parent's post but I don't think this time was on a route lookup if I'm reading the thread and post correctly the static file handler getting inserted multiple times. This handler will generally match on any route but then is doing something like "if file exists, return static file, if not look for the next handler" in this case the "if file exists" part was the "path check" thats taking…
A simple "if file exists" check shouldn't take 1ms on average.
OSes cache directory entries for a reason.
I mean, even Python manages 40,000 checks / second:
>>> timeit.timeit("os.path.exists(data)", setup="import os; import random; import string; data = os.path.join(r'C:\Windows\System32', ''.join(random.choice(string.ascii_letters) for _ in range(10)))", number=40000)
0.9998181355403517Re: Node.js in Flame Graphs
#224Earlier quoted context omitted.
You're the author of Haraka, a tool I'm prepared to try in production. This makes me _really_ worried that I'm making the right choice. (To explain: obviously C++ churns out machine code, the parent was talking about compiled vs uncompiled regexes – if I'm not terribly wrong, the compilation step is turning the regex into a finite automaton.)
V8 literally compiles regexps to X86 machine code the first time they are executed. They are not compiled into an FSA that gets walked in the traditional sense. Hopefully that lowers your concern level.
Doesn't that mean that you have exponential worst-case complexity?
Re: Node.js in Flame Graphs
#225Earlier quoted context omitted.
If the paths have some kind of non crazy Regex leading up to what gets munched for path variables (eg /path1 versus /path2) then you could at least build a tree of maps at each level, which would be roughly constant time for the most common cases.
> roughly constant time for the most common cases. For small values of n ;) A tree of maps would be consistently log(n), like any map. Even a hashtable would hash to buckets eventually, and are log(n).
Not if you're using a Cuckoo hashmap.
Honestly, I have no idea why people aren't taught Cuckoo-maps by default. It's simple, has true O(1) worst-case lookup, and has easy deletion. About the only problem is trying to prove insertion time.
Re: Node.js in Flame Graphs
#226> I can’t imagine how we would have solved this problem without being able to sample Node.js stacks and visualize them with flame graphs.
This has me scratching my head. The diagrams are pretty, maybe, but I can't read the process calls from them (the words are truncated because the graphs are too narrow). And I can't see, visually, which calls are repeated. They're stacked, not grouped, and the color palette is quite narrow (color brewer might help here?).
At least, I _can_ imagine how you could characterize this problem without novel eye-candy. Use histograms. Count repeated calls to each method and sort descending. Sampling is only necessary if you've got -- really, truly, got -- big data (which Netflix probably does), but I don't think the author means 'sample' in a statistical sense. It sounds more like 'instrumentation', decorating the function calls to produce additional debugging information. Either way, once you have that, there are various common ways to isolate performance bottlenecks. Few of which probably require visual graphs.
There's also various lesser inefficiencies in the flame graphs: is it useful (non-obvious) that every call is a child of `node`, `node::Start`, `uv_run`, etc.? Vertical real-estate might be put to better use with a log-scale? Etcetera, etc.
Re: Node.js in Flame Graphs
#227Earlier quoted context omitted.
They are arguments in favor of node.js being a reasonable tool for the job. > Java is both considerably more speedy for this sort of workload They're quoting 1ms latencies, that's pretty speedy. Maybe Java could do > there's a significantly larger pool of high end Java server developers compared to high end JavaScript server developers Oh, I would have thought there are far more JS* folks out there than Java. > Budge…
I'm not sure if I agree. Responding after 1ms is not the same (or even very much related to) routing a request taking up 1ms of cpu time. Although there are a ton of nuances that make the following a small oversimplification it does basically mean that purely for routing alone you have a hard cap on 1k lookups/sec/core. And building server software in any language takes a whole different set of skills and knowledge t…
The parent (and the article) are both talking about 1ms request latencies, not 1ms routing latency.
Re: Node.js in Flame Graphs
#228I wonder what the thought process was behind moving their web service stack (partially?) to node.js in the first place. For a company with the scale and resources of Netflix it's not exactly an obvious choice.
I share this thought, I'm not trolling, I really believe node is a bad solution for something like Netflix. Node has its perks but for a money making machine that relies solely on being available and providing a good customer experience, not so much. I can't imagine the ops nightmares at that size, one buggy code path and the entire cluster could be down. These are issues that drove me away from Node to Go, in my opi…
[deleted]
Re: Node.js in Flame Graphs
#229I read: "This turned out be caused by a periodic (10/hour) function in our code. The main purpose of this was to refresh our route handlers from an external source. This was implemented by deleting old handlers and adding new ones to the array" refresh our route handlers from an external source This is not something that should be done in live process. If you are updating the state of the node, you should be creating…
When I concluded what they had to be doing and then read the actual confirmation of what they were doing I was somewhat shocked. Why on Earth would you want to programatically recreate the routes in an express app?!?!? It would be really interesting to see a write up on what/why they think this kind of behavior is needed in the first place ....
Re: Node.js in Flame Graphs
#230Earlier quoted context omitted.
Of course there's a faster way! Combine all the routes into a DFA, then run the DFA over the URL. It's guaranteed to run in constant space and O(n) (n=URL length) time! The union of any set of regular languages is itself a regular language. You can use Ragel[1] to build your automaton. [1] http://www.colm.net/open-source/ragel/
This is the reason we need more people in JS land taking compiler courses and the like. There's a lot of tech out there that are speed sensitive yet do not apply the "best" solution to a problem solved in the 70s.
But then you need to seduce the JS runtime into actually performing your optimizations. Maybe time for C/C++ extensions to become more prevalent a la Python?