Live data from Hacker News

Node.js in Flame Graphs

techblog.netflix.com

201–210 of 259 posts

Re: Node.js in Flame Graphs

#201
post #165

Earlier quoted context omitted.

This. I wrote something that did this a few years ago. It took n patterns (not regex, simpler) and turned them into one DFA state table with a function ptr stored for each final state. Then it had a tiny runtime. Never thought of using it for route tables however. Nice idea. Edit: I did consider rewriting the runtime as a forth style interpreter as well but the time never appeared.

How long does it take to build the DFA when a new route is added? That seems like the major time performance problem point if new routes are added often or regularly.

I'm not sure I can think of a case where new routes should be added dynamically, is there ?

Re: Node.js in Flame Graphs

#202

Earlier quoted context omitted.

It's funny, my first job out of college was at a startup with an ex-Sun CTO, and he insisted on a single monolithic codebase. After he left I tried to organize an (aborted) project to modularize the codebase, since many of the other engineers ran into it as well. Having worked at Google in the interim, and having a number of acquaintances at Microsoft (which uses the multiple-repository approach) I can see the pros a…

>When you have a number of interdependent modules, then every change request and new feature has to go through that module's owner. If it's not their top priority (and it won't be), then getting your work done suddenly has a hard dependency on a team who is...generally pretty unresponsive, at least from your POV. Is that true? Again, I've never done development on that scale, so I could totally be wrong, but I'm seei…

Imagine what happens at scale if everyone forks the repo when a critical feature they need isn't being worked on by the core team. Suddenly you have dozens of forks. Now imagine that the core team eventually gets around to publishing that bugfix you really do want. All of these incompatible forks need to down-integrate the change and work around the little tweaks they've added. You have a mess.

Open source works because the projects that become popular can stake out a particular territory and say "This is our responsibility, this is your responsibility, these are the hooks you can tap into, and if you don't like it, fork or use a different project." Flip the perspective and imagine yourself as the user of an open-source project. There's always a huge amount of work you have to do to customize exactly what the framework gives you to what the product needs. This is okay, because that's the bargain you know you're signing up for when you use the open-source framework, and it is after all what you're getting paid to do. If you don't believe it saves you more effort than it costs you, then don't use it. (In fact, there's a huge graveyard of projects on GitHub where the author tosses their personal way of doing things over the wall and then wonders why nobody uses it or contributes to it. This is why: the benefits don't outweigh the costs.)

Now imagine that you're at a big corporation. There is a lot of work that somebody has to do to make the product work right. All of you are getting paid by the same entity, the company. Who does the work? The team who wants it? The team whose codebase it's in?

Open source projects have the luxury of defined interfaces; they get to set the boundaries of what their code does themselves, and can turn away customers who need more than they provide. Internal teams do not: if they say no to a critical feature (regardless of how different it is from their existing interface), the product doesn't ship, and then an executive gets mad. So often adding features is non-negotiable, the features are significant enough that there's a real risk of impacting the stability or performance of the core product - and yet it's still the right thing for the business to do. The big problem is that each individual team wants clean code and well-defined interfaces, yet the benefit accrues to the company as a whole...hence, resentment between teams about who's got to spend significant work mucking up their pristine design.

Re: Node.js in Flame Graphs

#203

Earlier quoted context omitted.

Actually, open source allows something that is not possible with commercial software. You don't have to read all of the code and understand all of the implementation details, but you have that option if you need to. I think having the option is a great benefit as we can see from this example. When they started debugging they could reference the express source and figure out what was going on. If they were using some…

I agree with you that the greatest strength of open source is that you can just go read the code and (optionally) fix it. Netflix did this and it got them past their crisis. I think you missed my point though, which was that there isn't any sort of fitness function being applied to open source. In the closed source environment that it 'price' (caveat walled gardens). By paying for the software customers "vote with th…

There totally is a fitness function for open-source. People who find that an open-source program creates more hassles than it solves don't use it. Folks who blog about their hassles induce other people not to use it.

There are a number of open-source projects - web.py, Mongo, Angular, Ember - that I am actively avoiding because IMHO their benefits to me don't outweigh risks or previous hassles experienced. Much better to use tried-and-true open source projects like MySQL, Postgres, vanilla JS, and Django that I have had good experiences using. There are many others like Polymer or Docker that I am intrigued by, would love to give a try, but am sitting on the sidelines until other people uncover the most obvious pitfalls.

Economically, open-source projects are just startups where the monetary cost of use is zero. That doesn't mean that the total cost of use is zero, and as you're evaluating potential products, you need to be careful to figure in time expenditure spent debugging as a cost. But you are incredibly naive if you believe that just because you are paying a million dollars for a piece of software, you will spend less time debugging and integrating it than a well-used piece of open-source software. Hell, I once was the college intern writing that million-dollar piece of software. All it takes to sell something is chutzpah.

Re: Node.js in Flame Graphs

#204

Crazy talk. In 1ms, I can perspective transform a moderately big image. NodeJS cant iterate through a list. We really need a 60 fps equivalent for web stuff. You have 16ms, thats it.

FWIW, a lot of the stuff the Chrome+Polymer team is working on explicitly has a 60fps goal and 16ms frame budgets. I remember in my last project at Google, I spent a lot of time working with the Chrome team to get various parts of websearch rendering under the 16ms budget. (I couldn't manage it given the amount of legacy code we had to work with, but various other people have continued the work since I left, so hopefully they've had more luck.)

Re: Node.js in Flame Graphs

#205
post #165

Earlier quoted context omitted.

How long does it take to build the DFA when a new route is added? That seems like the major time performance problem point if new routes are added often or regularly.

I'm not sure I can think of a case where new routes should be added dynamically, is there ?

That's the exact thing that caused Netflix's problem. Read the article.

Re: Node.js in Flame Graphs

#206
post #181

Earlier quoted context omitted.

I'm not disagreeing and that's a perfectly valid reason to go for node for smaller projects or for prototyping. What many people fail to realise is that the "concurrency" model (it is solved by simply not having concurrency) isn't a magic bullet. It comes with large drawbacks. Not a lot of things are complicated because people like it that way. Some things are as complicated as they need to be to be used effectively.…

An example of basic concurrency I have great difficulty doing correctly in Java: n = 0; avg = 0; function twothings() { thing1(function(ret) { ++n; avg += (ret - avg)/n; }); thing2(function(ret) { ++n; avg += (ret - avg)/n; }); } I want to dispatch two I/O operations to run concurrently and have their completions modify some shared state in a serializable fashion, which can be read at any stage in the process and hav…

[deleted]

Re: Node.js in Flame Graphs

#207
post #181

Earlier quoted context omitted.

I'm not disagreeing and that's a perfectly valid reason to go for node for smaller projects or for prototyping. What many people fail to realise is that the "concurrency" model (it is solved by simply not having concurrency) isn't a magic bullet. It comes with large drawbacks. Not a lot of things are complicated because people like it that way. Some things are as complicated as they need to be to be used effectively.…

An example of basic concurrency I have great difficulty doing correctly in Java: n = 0; avg = 0; function twothings() { thing1(function(ret) { ++n; avg += (ret - avg)/n; }); thing2(function(ret) { ++n; avg += (ret - avg)/n; }); } I want to dispatch two I/O operations to run concurrently and have their completions modify some shared state in a serializable fashion, which can be read at any stage in the process and hav…

What I think you're missing is that in JS, this is not being run concurrently. Existing JS runtimes are single-threaded—if you actually want to take advantage of multiple CPUs you must run more than one process. It's very easy not to have concurrency bugs when there's no concurrency.

In Java, you have access to real threads and can actually run two pieces of code concurrently. This introduces a lot of potential issues, but also allows you to take advantage of multiple cores.

If all you want is non-blocking IO (what node gives you), there are a handful of great libraries that provide that on the JVM.

Re: Node.js in Flame Graphs

#208

Earlier quoted context omitted.

Yes, but their original bug was from dynamically loading routes from an external source. I don't see how Express is to blame for this. Moving to Restify is not a solution, but they state having different reasons for moving (support for bunyan logging? But Express already supports this too).

the bug was related to dynamically loading routes, but the true cause was that express allowed duplicate handlers. They were loading routes dynamically correctly, that wasn't a problem, it was that when doing that express let them duplicate routes.

That's a feature.

From the API docs:

> Multiple callbacks may be given; all are treated equally, and behave just like middleware. The only exception is that these callbacks may invoke next('route') to bypass the remaining route callback(s). This mechanism can be used to perform pre-conditions on a route, then pass control to subsequent routes if there's no reason to proceed with the current route.

Re: Node.js in Flame Graphs

#209
post #38
post #19

Earlier quoted context omitted.

Nope. You see an option for a plural here? http://www.merriam-webster.com/dictionary/learning

Dictionaries are descriptive , not prescriptive . They can only tell you if something is a word, not whether something isn't a word.

Well certainly there is no law against using made-up words. If that's what you want to do, have at it.

At least it will help people get their buzzword bingo cards filled up sooner.

Post reply on HN