Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical challenges and misreadings of literature create unreplicable science [pdf]

stat.columbia.edu

Post reply on HN
Print view View original

11–20 of 57 posts

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#11

Post by light_hue_1 »

littlestymaar wrote: post #9
mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Economics works exactly like this, but at a fractal level: there are many different subfields living in the grand family of economists which barely communicate with one another, often have dramatically different theories that lead to opposing conclusion, none of which even trying to give an accurate description of the world but instead promote one particular ideological bias. Every once is a while someone from one ch…

This happens everywhere. Let me tell you a story about who studies language and how.

Linguistics studies language right? It has a few hundred year history of doing so seriously. But once AI people wanted to also have a look linguists became unhappy. They didn't like the idea of someone else saying something about their problem with a different set of tools. Those tools don't provide explanations that they approve of. Even the thought of measuring things over a dataset to check if your linguistic theories work made them bristle with Chomsky famously rejecting the idea of evidence at scale.

So AI people created natural language processing. Their own field to study the same problem. One in a while someone tried to do something to bridge the gap which led to the famous "every time I fire a linguist my performance goes up".

Many people in NLP see linguists as outdated fossils. Most would probably vote to just shut down linguistics departments I suspect.

Then something funny happened. The LLM people did to NLP what they did to linguistics. And the NLP community rejected them. Because LLMs don't provide an explanation that they approve of. Recently just like in linguistics the President of ACL, the main NLP organization, said recently that NLP is not about machine learning! The absurdity of that statement just boggles the mind. So the LLM people now have their own conference and field.

A lot of people cannot accept that things must change over time. They want to do research the way they did it 10 or 40 years ago. So they create an immune response. It's human nature.

As someone who doesn't feel this way and enjoys moving forward, I'm really sad about what's happened to NLP in the past few years. It would be nice if we could stop this rift. But it won't happen with the current hate filled President. I stopped sending any work there until they reform.

Just the cycle of academic life. I wonder if science would move faster if we could select for people who are willing to change with the times.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#12

Post by mjburgess »

light_hue_1 wrote: post #11
littlestymaar wrote: post #9

Earlier quoted context omitted.

Economics works exactly like this, but at a fractal level: there are many different subfields living in the grand family of economists which barely communicate with one another, often have dramatically different theories that lead to opposing conclusion, none of which even trying to give an accurate description of the world but instead promote one particular ideological bias. Every once is a while someone from one ch…

This happens everywhere. Let me tell you a story about who studies language and how. Linguistics studies language right? It has a few hundred year history of doing so seriously. But once AI people wanted to also have a look linguists became unhappy. They didn't like the idea of someone else saying something about their problem with a different set of tools. Those tools don't provide explanations that they approve of.…

The goal of physics isn't to produce video games -- that a video game can generate a frame using unphysical formulae and unphysical processes (eg., rasterisation) does not invalidate physics.

There's linguistics which produces descriptive theories of the practice of language. There's "computational linguistics" which aims to build discrete algorithmic models of language. Both of these are interested in explanations of actual physical phenomena.

Machine Learning is largely an engineering discipline which, like video games, can generate output using any method which will convince a human user. Indeed, often more radically, ML works by replaying variations of past convincing output rather than have any explicit simulation involved at all.

Thus there's no sense in which LLMs supercede linguistics, nor somehow computational linguistics superseding this.

This would be like claiming rasterisation means we should retire optics and the physics of light.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#13

Post by michaelt »

blackeyeblitzar wrote: post #2

On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.

> For example political polling, data point in time surveys, census, etc. Without verification, what good is the data?

Political polls are regularly wrong https://www.bbc.co.uk/news/articles/cj4ve004llxo

In my country, supporters of one political party seem a lot more likely to tell polling companies they're "undecided" - so the pollsters try to adjust their predictions, based on what they've seen in previous elections.

But the problem is getting the adjustment right is very difficult. The political landscape can change a great deal between elections. If John McCain and Donald Trump appeal to very different sets of voters, why should voter behaviour observed with McCain hold true with Trump?

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#14

Post by pacman001 »

mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Every single time (some 4 times in my past life) when I’m familiar with/close to the background story of something that played out in the news I come to the same conclusion: news doesn’t report the facts. I’m in Western Europe, btw. I advise to run this experiment yourself.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#15

Post by GPerson »

light_hue_1 wrote: post #11
littlestymaar wrote: post #9

Earlier quoted context omitted.

Economics works exactly like this, but at a fractal level: there are many different subfields living in the grand family of economists which barely communicate with one another, often have dramatically different theories that lead to opposing conclusion, none of which even trying to give an accurate description of the world but instead promote one particular ideological bias. Every once is a while someone from one ch…

This happens everywhere. Let me tell you a story about who studies language and how. Linguistics studies language right? It has a few hundred year history of doing so seriously. But once AI people wanted to also have a look linguists became unhappy. They didn't like the idea of someone else saying something about their problem with a different set of tools. Those tools don't provide explanations that they approve of.…

There’s no single forward direction in science. We’re in a machine learning hype cycle. That doesn’t mean we must fire everyone in every other department.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#16

Post by HPsquared »

mjburgess wrote: post #12
light_hue_1 wrote: post #11

Earlier quoted context omitted.

This happens everywhere. Let me tell you a story about who studies language and how. Linguistics studies language right? It has a few hundred year history of doing so seriously. But once AI people wanted to also have a look linguists became unhappy. They didn't like the idea of someone else saying something about their problem with a different set of tools. Those tools don't provide explanations that they approve of.…

The goal of physics isn't to produce video games -- that a video game can generate a frame using unphysical formulae and unphysical processes (eg., rasterisation) does not invalidate physics. There's linguistics which produces descriptive theories of the practice of language. There's "computational linguistics" which aims to build discrete algorithmic models of language. Both of these are interested in explanations o…

That's a nice dividing line, science and engineering are different for sure. But science makes use of engineering knowledge and engineering makes use of scientific knowledge.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#17

Post by HPsquared »

blackeyeblitzar wrote: post #2

On page 11 there is a mention of taking the result of self reporting (surveys) at their word. I’ve wondered about this issue not just in science but other situations. For example political polling, data point in time surveys, census, etc. Without verification, what good is the data? And yet you often see such self reported data quoted by articles or papers as if it were factual.

It's like the uncertainty principle: even the act of asking someone a question will likely change their views.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#18

Post by light_hue_1 »

mjburgess wrote: post #12
light_hue_1 wrote: post #11

Earlier quoted context omitted.

This happens everywhere. Let me tell you a story about who studies language and how. Linguistics studies language right? It has a few hundred year history of doing so seriously. But once AI people wanted to also have a look linguists became unhappy. They didn't like the idea of someone else saying something about their problem with a different set of tools. Those tools don't provide explanations that they approve of.…

The goal of physics isn't to produce video games -- that a video game can generate a frame using unphysical formulae and unphysical processes (eg., rasterisation) does not invalidate physics. There's linguistics which produces descriptive theories of the practice of language. There's "computational linguistics" which aims to build discrete algorithmic models of language. Both of these are interested in explanations o…

That's like saying that it's not astronomy if I'm using a radio telescope, only optical telescopes are real astronomy. It makes zero sense. The tools are irrelevant. We don't have physics with and without ml journals.

It's like saying AlphaFold isn't biology because it uses ml. It's absurd

One goal of linguistics is to produce an account of how language works. If you understand how something works you can reproduce it. This was always a core part of linguistics. Just look at Chomsky's work. Or at what counts as evidence in a linguistics paper. It is a model that "explains" some feature of language.

NLP does exactly the same thing as linguistics. It produces explanations and makes predictions about language. But it does so with ML.

I think you're confused about optics and rasterization. There are countless ways to relate the two. Rasterization is not unphysical, no more so than say, finite element methods are unphysical. Heck physics is in the name of PBR. It's an approximation. And a fine one that people actually use in both science and games.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#19

Post by littlestymaar »

pacman001 wrote: post #14
mjburgess wrote: post #8

We're increasingly aware today of how the media operates cycles of self-referential and self-justifying citations: a TV show will quote an article that reports "some people" taking an issue, which ends up being a quote from someone interviewed for another newspaper article.. and so on. This "legitimacy laundering" is rampant, and we're now getting towards media literacy levels which expose it for many people. However…

Every single time (some 4 times in my past life) when I’m familiar with/close to the background story of something that played out in the news I come to the same conclusion: news doesn’t report the facts. I’m in Western Europe, btw. I advise to run this experiment yourself.

That's the problem when you have people responsible for news having the business model of ad-sposored entertainment and not fact reporting.

Re: Statistical challenges and misreadings of literature create unreplicable science [pdf]

#20

Post by ykonstant »

Besides the sociological problems listed, we must always be conscious of how counterintuitive and difficult statistical inference itself can be. Good things to search for are statistical fallacies, probabilistic paradoxes and books like Counterexamples in Probability.

And it is not sufficient to read about them once or twice; researchers who use statistical inference regularly must revisit these caveats at least as regularly.

Myself, I have taught Probability and Statistics many times, discussed and dispelled many misconceptions by students. Would I be 100% sure I will not be caught up in a fallacy while informally thinking about probability? I wouldn't even be 10% sure; any intuition I conjure up, I would triple check as rigorously as possible.

Post reply on HN