Live data from Hacker News

The Lindy Effect

arxiv.org

81–88 of 88 posts

Re: The Lindy Effect

#81
post #9

I like to use the Lindy effect as an everyday heuristic, quite often for choosing what books to read. That is: if a book has been highly relevant, say, for the past 20 years, it may have considerable relevance 20 years from today as well. Conclusion: I will think about buying that physical book for my home library, so my children would have a carefully-filtered collection of "classics" from many eras by the time they…

> so my children would have a carefully-filtered collection of "classics" from many eras by the time they reach young adulthood.

I have that. My three adult children read none of them.

Re: The Lindy Effect

#82
post #55

I really wish that we, software developers, apply Lindy rule while choosing library/framework/language/SaaS tool etc., Lot of developers underestimate the value of collective brain power that has gone into making a system mature, not to talk about community and ecosystem support. I was extremely hesitant to get onboard Golang bandwagon 5-7 years ago. I reluctantly said OK to try it out in a non critical service and a…

Care to elaborate on the Go issues you encountered? Nowadays Go is very well established, unless you specifically meant that the issues happened 5-7 years ago.

Of the top of my head:

1) Lack of libraries.

2) Little to non-existent tooling & observability support.

3) Lots of challenges around gRPC. That it needed a discovery service such as etcd was a big source of pain. I remember two or three massive downtimes where an adjacent team went through hell for a day just to get back a high throughput system back online. gRPC + etcd was very susceptible to thundering herd problem.

4) Lack of experienced developers who would guide others idiomatic way of doing things. And related, go routines need a different mindset and somewhat a grounds up re-learning. The code would routinely get into deadlock sort of scenarios.

I conceded that most of the above are not inherent to Go but to any new system/language. I'm sure they all have been addressed now as the language and ecosystem has matured. That was the point I was trying to make though; don't jump onto a new bandwagon. Of course you want to learn it on your own and experiment that's totally fine but to use it in production is fraught with risk IMO.

Re: The Lindy Effect

#83
post #29
post #16

It seems like the Lindy effect for a positive real number valued ramdom variable X, is the following identity E[X | X >= a] = 2 * a

> E[X | X >= a] = 2 * a Is it possible to derive the maximum entropy probability distribution subject to that constraint?

Let f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable.

We can write the identity as

    int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a.
We rewrite this as

    int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx.
Taking the derivative with respect to a on both sides yields

    - a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)).
Taking another derivative yields

    - f(a) - a f'(a) = - 2 f(a) - 2 f(a) - 2 a f'(a).
We rewrite this as

    f'(a) = - 3 f(a) / a,
and furthermore

    f'(a) / f(a) = - 3 / a.
We can write this as

    (ln ○ f)'(a) = - 3 / a.
As far as I can tell, we can not find a solution on all of [0, oo). Let's assume that failure is impossible up to an early time t. This means that we assume f(x) = 0 for all x 0, meaning we have a jump discontinuity in the probability density at x = t.

We integrate from t to x

    int_t^x (ln ○ f)'(a) da = int_t^x - 3 / a da.
This evaluates to

    ln(f(x)) = ln(f(t)) - int_t^x 3 / a da
             = ln(f(t)) - 3 ln(x) + 3 ln(t)
             = ln(f(t)) - ln(x^3) + ln(t^3)
             = ln(f(t) t^3 / x^3).
In particular

    f(x) = f(t) t^3 / x^3.
Right now f(t) is still a free parameter, but recall that

    1 = int f(x) dx
      = int_t^oo f(t) t^3 / x^3 dx
      = f(t) t^3 [- 1 / (2 oo^2) + 1 / (2 t^2)]
      = f(t) t / 2
      
has to hold, so we find that

    f(t) = 2 / t
and

    f(x) = 2 t^2 / x^3.
So these are the probability densities that fulfill the "modified" Lindy effect, where failure is impossible before time t > 0.

I haven't double checked this so maybe there is a mistake, but I'm quite convinced that this line of reasoning leads to a unique distribution for each t > 0 and impossibility when t = 0.

Re: The Lindy Effect

#84
post #83
post #29

Earlier quoted context omitted.

> E[X | X >= a] = 2 * a Is it possible to derive the maximum entropy probability distribution subject to that constraint?

Let f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable. We can write the identity as int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a. We rewrite this as int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx. Taking the derivative with respect to a on both sides yields - a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)). Taking another derivative yields - f(a) -…

So the Pareto distribution with alpha=2 produces the Lindy effect. Nice result! It looks like this generalises to E(X|X>a)=ka (with alpha depending on k).

Re: The Lindy Effect

#85
post #66

Earlier quoted context omitted.

Every page of the KJB seems to have a phrase that has entered the language (e.g. give up the ghost) - though I can't be sure that those phrases weren't already popular and merely recorded.

Umm, no. There is a huge selection bias at work here. Most people, when they read the bible, read Genesis and Exodus, the Gospels, Psalms, Ecclesiastes and a couple of other books. It is true that lots of phrases, images, stories, quotations and so on from those heavy-hitting parts of the Bible have made their way into the general culture. The vast majority of the Bible is not at all influential on either our religio…

You clearly are more informed than me, but my experience was from randomly reading pages. Maybe I got lucky. Though I recall one page with long lists of ancestry, x begat y - similar to a telephone book as you mention.

Re: The Lindy Effect

#86
post #84
post #83

Earlier quoted context omitted.

Let f: R -> R be the probability density function of X. It fulfills int f(x) dx = 1 and we furthermore assume that f is differentiable. We can write the identity as int_a^oo x f(x) dx / int_a^oo f(x) dx = 2 a. We rewrite this as int_a^oo x f(x) dx = 2 a int_a^oo f(x) dx. Taking the derivative with respect to a on both sides yields - a f(a) = 2 int_a^oo f(x) dx + 2 a (- f(a)). Taking another derivative yields - f(a) -…

So the Pareto distribution with alpha=2 produces the Lindy effect. Nice result! It looks like this generalises to E(X|X>a)=ka (with alpha depending on k).

I didn't notice that is a Pareto distribution! Well, that is quite nice indeed.

Re: The Lindy Effect

#87
post #56

Earlier quoted context omitted.

Also, we are too kind to some classics out of nostalgia perhaps. I've been greatly disappointed by top rated classic spy and sci-fi novels. The kind of storytelling people got away with 50 years ago really doesn't work anymore today. Think of bad pacing, characters without depth, lazy stereotyping, deus ex machinas. So on the one hand recency bias results in us giving undue attention to new books. On the other hand w…

Yeah I tried reading Solaris and was laughing at how bad it was. Felt the same way about nueromancer. Had a good run of trying out sci fi classics and being dissapointed. Philip k dicks stuff was solid tho. Ubiq is the shit

I'm also having a hard time with neuromancer, but I'm afraid that Gibson did this on purpose

Re: The Lindy Effect

#88

Earlier quoted context omitted.

You overfit for classics and skip on maybe very entertaining contemporary reads. Feels weird to me to hipster out on the "I build a library of classics only". With this approach you'd never read Dark Matter, which is one of my favorites in the past few years. I also don't care about rankings, top sellers etc. etc., but contemporary stuff is still worth the read. Feels very weird to me tbh.

The fact that you mention Dark Matter (no hate, taste is subjective etc.) as potentially rivaling actual classics is for me just further proof that fighting recency bias and hype is important. But yes, of course some modern books will end up standing the test of time as well. But it's hard to judge that a priori without actually reading the book yourself. You will spend a finite amount of time reading in your life. I…

It's not about dark matter rivaling "classics" (which we established are shit enough most of the times anyway). And if you're US American, your classics are most probably not my classics.

> an older book is a much safer bet if you want to maximize the value of your limited time

I 100% disagree with that. Value is highly subjective. I've read a lot of the books that are considered as "the most important books from the western hemisphere" and let me tell you, most of it is garbage. It's just a classic because back in the day it was novel and exciting, nothing else like it. You can have a good time with both, modern and old books. It's about your selection process, not the age of the book.

Reading has multiple purposes. Entertainment and education, most importantly. So I read books depending on the goal I want to achieve.

PS: if you read the commercial garbage from reading lists and what's new, you'll have a bad time, granted. As everybody can write a book nowadays, quality is harder to find tho.

Post reply on HN