Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The eigenvector of “Why we moved from language X to language Y”

erikbern.com

Post reply on HN
Print view View original

151–160 of 193 posts

Re: The eigenvector of “Why we moved from language X to language Y”

#151

Post by MrQuincle »

omaranto wrote: post #144
erikbern wrote: post #125

Earlier quoted context omitted.

Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay. I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best…

That bit about the stationary distribution not changing if you add a diagonal matrix sounds completely wrong to me. Let me see if I understand what you mean. Given a matrix M with non-negative entries (and no row of just zeros), let S(M) denote the stochastic matrix you get by normalizing each row of M. You are saying that if M is any matrix and D is a diagonal matrix with non-negative entries then S(M) and S(M+D) ha…

Moreover data is collected over the entire history. A matrix is a linear operator from time step T_i to T_i+1. By conflating all historical observations into one matrix it definitely is not an ordinary transition matrix.

That apart from the fact that it is questionable that it can be represented by an operator that is finite and linear.

It's more likely a stochastic process (infinite matrix) with births and deaths.

I would be surprised if it became true. :-)

Re: The eigenvector of “Why we moved from language X to language Y”

#153

Post by thaumasiotes »

j2kun wrote: post #135
erikbern wrote: post #125

Earlier quoted context omitted.

Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay. I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best…

The thing I like most about this post is that it's falsifiable. We will know in ten years whether C and Java are still popular, and whether Go succeeds in the sense this data suggests. So thank you for being concrete and clear, even if it's all in fun and other people don't like it :)

> whether Go succeeds in the sense this data suggests

An interesting thing about this methodology is that it is extremely sensitive to the age of a language. It's possible to switch from an old language to a new language, but not the other way around -- so if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects on it and subsequently gotten sick of it, the future distribution by this method can only be 100% New Language. (Because sometimes people switch to New Language, but no one ever switches away.)

Actually, to predict the future distribution of language use, you also need to know the rate of people moving from nothing ("I just had a brilliant idea!") to each language. If everyone eventually transitions to Go, but everyone starts in Ruby, then the division of market share between Go and Ruby depends in part on how frequently people start new projects.

Re: The eigenvector of “Why we moved from language X to language Y”

#154

Post by jlrubin »

mathgenius wrote: post #37
jlrubin wrote: post #13

I appreciate that the author wanted to implement their own eigen vector/value method, but really they should use: numpy.linalg.eig(x)[0] numpy.linalg.eigvals(x)[0] Numerical stability can be hard to get right...

Yes, but... The matrix has all non-negative entries, and the author is after the highest eigenvalue/vector so I think this means stability is just not an issue. The only possible issue is time to convergence. The nice thing about the power method is its conceptual simplicity. In cases like this, it's quite hard to screw it up. And it will scale far beyond those numpy functions (not that this is needed for this exampl…

That's not really the definition of numerical stability. You're thinking of linear system stability, which is a whole other topic. Numerical stability is how resilient a computation is to computational error. These computations are taking place using floating point values, the issues arise in the error terms for floats. The author's algorithm is reproduced below:

    m /= m.sum(axis=0)[numpy.newaxis,:]
    u = numpy.ones(len(items))

    for i in xrange(100):
        u = numpy.dot(m, u)
        u /= u.sum()
Immediately concerning is the use of a dot product and a sum, which will lose a lot of information when the values being added are of different orders of magnitude. (For example `M + epsilon = M` in many floating point computations).

Also, while scalability for this size computation is way overkill, it is precisely a problem where the numerical stability problem gets even worse. Imagine if I have data on some kind of power law, and compute left-to-right `M+epsilon_0+...+epsilon_n`. No matter now large `n` is, for sufficiently different order of magnitude M and epsilon all the information is lost -- `epsilon_0+...+epsilon_n+M` could be an entirely different number. Highly recommend checking out the LAPACK stability guide here http://www.netlib.org/lapack/lug/node72.html for more.

Re: The eigenvector of “Why we moved from language X to language Y”

#156

Post by erikbern »

ArneBab wrote: post #148
unpythonic wrote: post #143

Earlier quoted context omitted.

The contingency table shows these very clearly, but I believe the question is about the future popularity table. The future probability table shows that Swift is to the left of (smaller future probability) of Objective C. The coloring of the chart shows a much darker square for Swift -> Objective C than for Objective C -> Swift. It seems surprising given your contingency table that your analysis would show that Objec…

if I understand it correctly that means that most people who move away from swift move to Objective C, but people moving away from Objective C also move to C# and Java.

Yes, this is correct. The second table shows conditional probabilities. Almost everyone moving from ObjC goes to Swift, but conversely out of the people moving from Swift it's more spread out

Re: The eigenvector of “Why we moved from language X to language Y”

#157

Post by j2kun »

thaumasiotes wrote: post #153
j2kun wrote: post #135

Earlier quoted context omitted.

The thing I like most about this post is that it's falsifiable. We will know in ten years whether C and Java are still popular, and whether Go succeeds in the sense this data suggests. So thank you for being concrete and clear, even if it's all in fun and other people don't like it :)

> whether Go succeeds in the sense this data suggests An interesting thing about this methodology is that it is extremely sensitive to the age of a language. It's possible to switch from an old language to a new language, but not the other way around -- so if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects…

I didn't interpret the results in this post as "predicting the future distribution of language use." Rather, I interpreted the ranking as an indicator of a qualitative trend in that distribution. I think the author also made this very clear.

And of course, all things trend toward newness, so your objection there seems more about time or human psychology than the methodology of the post.

Re: The eigenvector of “Why we moved from language X to language Y”

#158

Post by peterwwillis »

lmm wrote: post #59
peterwwillis wrote: post #20

Earlier quoted context omitted.

C is a tadpole in the ocean of easily preventable bugs.

Then why do we see a major internet security bug that would be simply impossible in any other language every couple of months?

https://www.cvedetails.com/vulnerability-list/year-2016/mont...

Note the types of vulnerabilities and the languages they use. DoS, File Inclusion, XSS, Exec Code, Dir Traversal, Priv Escalation, SQLI, Bypass, CSRF, Info leak, etc.

Many of them (including ones in C) have nothing to do with memory protection. Those that do (null pointer deref, use-after-free, memory leak, buffer overflow, etc) are all trivially protected with small kernel patches that have been around for 17 years, and of course most of these vulns would become trivial with proper mandatory access control. But for reasons that completely escape me, nobody has adopted these basic techniques to prevent small bugs from becoming big holes.

Now balance the common holes in C against all the other bugs in higher level languages with otherwise suitable memory protection, and consider that at least with C there are basic steps that prevent many of these from becoming problems, whereas with other languages you need a hodge-podge of different, more complicated countermeasures. C is actually easier to secure because its bugs are common and not difficult to catch by the kernel.

Honestly, if people spent as much time developing new industry best practices for use of the language as they do complaining about it, this would be a non-issue. But C isn't trendy, so let's all crap on it and pretend it's the only issue so we can have fun reinventing classical bugs with new languages.

Re: The eigenvector of “Why we moved from language X to language Y”

#159

Post by noir_lord »

Maro wrote: post #127

I wish more people would read 'Hack and HHVM', written by Owen Yamauchi, formerly a member of Facebook’s core Hack and HHVM teams. http://shop.oreilly.com/product/0636920037194.do The hidden lesson for me was that rewriting the code in is not the only option. Another option is to slowly improve the language/runtime itself until you've essentially switched it out underneath the application, which is what happened at F…

As an unintended side-effect (maybe), the mere existence of HHVM acted as a spur to PHP generally, things have improved radically in PHP-land over the last few years.

Re: The eigenvector of “Why we moved from language X to language Y”

#160

Post by badthingfactory »

rjbwork wrote: post #60

Highly skeptical of so many people migrating from C# to C, or python to Matlab, to give a couple of examples. This seems like a highly flawed methodology from many perspectives, as pointed out in comments.

Also C# to VB. That's pretty much a one-way street in .NET development. I have yet to meet a developer interested in moving from C# back to VB.
Post reply on HN