What's wrong with social science and how to fix it • Hacker News Board

Re: What's wrong with social science and how to fix it

#101

Post by hinkley » Mon, Sep 14, 2020, 4:34 PM UTC

doonesbury wrote: post #20

hinkley wrote: post #5

Earlier quoted context omitted.

If someone is selling you on rethinking an entire discipline they are probably overstating their case a bit. Or you’re ignoring them outright. That doesn’t mean they’re wrong, necessarily. Overcoming inertia is a huge challenge. Daunting, even.

Agree. In the face of institutional inertia - here crossing industry and academic fields - your starting point is when I see a fly I use a canon. It is extremely difficult to budge. As so the usual reminder: the first 51% of communication is repetition

I have found that the repetition usually only works if it comes from multiple sources. It's the primary reason I encourage the 'new guy' to take their "this doesn't make sense" questions up the chain of command or to the groups we collaborate with.

One, I hate to crush a spirit, although I'm just sending them to someone else to do it. Two, about one time in ten they come back with a preliminary roadmap to a solution.

I can tell Steve until I'm blue in the face that this code is nuts and get nowhere, or I can send three other people to tell him once and finally he'll take it seriously.

I don't think it's malicious, I think it's a combination of basic human psychology with learning strategies. Sometimes you have to shop for new instruction because the perspective your teacher brings simply doesn't resonate. Each person reinforces the neural connections and phrases it a little differently (perhaps especially with new people, because they are fresh off the street and don't use our jargon yet?)

Re: What's wrong with social science and how to fix it

#102

Post by boomboomsubban » Mon, Sep 14, 2020, 7:51 PM UTC

disown wrote: post #100

mkatx wrote: post #97

Earlier quoted context omitted.

Math is the foundation of science, right? Things in science are provable because of the rigorous mathematical proofs applied to data, correlating with reproducible results, is my perception. There are many models of societies currently and previously in existence that can be measured, compared, and correlated with predictable cause and effects. Society is not constant, it's probabilistic, like many topics in physics…

> Math is the foundation of science, right? No. The scientific method is the foundation of science. Math is a wonderful tool that is used in science and other fields like business, architecture, music, etc. > Things in science are provable because of the rigorous mathematical proofs applied to data, correlating with reproducible results, is my perception. Nothing in science is "provable". Science is not math or logic…

>No. The scientific method is the foundation of science

Simple question then, which aspect of the scientific method is not compatible with "human law?" I can develop a question about how society works, form a hypothesis, develop experiments, and analyze the results.

I may develop poor experiments and as a result only be able to provide limited analysis permanently tainted by a "human law" bias, but it would still be science.

Re: What's wrong with social science and how to fix it

#103

Post by pas » Tue, Sep 15, 2020, 12:39 PM UTC

thu2111 wrote: post #99

pas wrote: post #95

Earlier quoted context omitted.

Huh? That sounds exactly the thing that a ML system would learn quickly from the data. You probably don't even need shiny deep learning (though it helps). Just like with the Netflix Prize stuff, where the conclusion was very similar, ie. just dump in as much data as you can, crank up the ML machinery, and it'll discover the features (better than you can engineer them) and learn what to use for recommendation ranking.…

Even GPT-3 doesn't encode common sense, which is why it can't do a lot of basic physical reasoning. It's "just" word prediction, albeit very impressive word prediction. If you look at what GPT produces closely, a lot of it is simply bullshit. It sounds plausible but is wrong. That's exactly the wrong type of AI to detect plausible-but-wrong-bullshit papers, which are the most common type.

Right, I worded that a bit lazily. There's no confidence score output from GPT-3, but if there were and if the user would select to only get high confidence results then it would shut up quickly. And that's what I meant by common sense. Of course it depends on the corpus. It's really-really just text, as you said. (It's possible that it can somehow eventually encode high level things like arithmetic, but so far it seems, even if it does have that model somewhere embedded, it doesn't know how/when to use it.)

The language model (GPT-3) doesn't have to understand physics, it just have to help extract out some semantics of the paper.

There needs to be a classifier on top trained with a labeled set of good and bad papers.

Re: What's wrong with social science and how to fix it

#104

Post by thu2111 » Wed, Sep 16, 2020, 7:56 AM UTC

pas wrote: post #103

thu2111 wrote: post #99

Earlier quoted context omitted.

Even GPT-3 doesn't encode common sense, which is why it can't do a lot of basic physical reasoning. It's "just" word prediction, albeit very impressive word prediction. If you look at what GPT produces closely, a lot of it is simply bullshit. It sounds plausible but is wrong. That's exactly the wrong type of AI to detect plausible-but-wrong-bullshit papers, which are the most common type.

Right, I worded that a bit lazily. There's no confidence score output from GPT-3, but if there were and if the user would select to only get high confidence results then it would shut up quickly. And that's what I meant by common sense. Of course it depends on the corpus. It's really-really just text, as you said. (It's possible that it can somehow eventually encode high level things like arithmetic, but so far it se…

I think there is a confidence score actually! Most blogs about it don't show them but this one went into it:

https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/

It's really cool how the uncertainty prompts alter the confidence associated with the next words.

I guess I'm not disagreeing with you in the abstract that a theoretically strong enough AI could identify bad papers, especially if it had some help for 'real' arithmetic. It at least could flag the most basic issues like plagiarism, cited documents that don't contain the cited fact, etc. Detecting claims that are themselves implausible seems like the hardest task possible, however. That's very close to general AI.

Re: What's wrong with social science and how to fix it

#105

Post by pas » Thu, Sep 17, 2020, 2:51 PM UTC

thu2111 wrote: post #104

pas wrote: post #103

Earlier quoted context omitted.

Right, I worded that a bit lazily. There's no confidence score output from GPT-3, but if there were and if the user would select to only get high confidence results then it would shut up quickly. And that's what I meant by common sense. Of course it depends on the corpus. It's really-really just text, as you said. (It's possible that it can somehow eventually encode high level things like arithmetic, but so far it se…

I think there is a confidence score actually! Most blogs about it don't show them but this one went into it: https://arr.am/2020/07/25/gpt-3-uncertainty-prompts/ It's really cool how the uncertainty prompts alter the confidence associated with the next words. I guess I'm not disagreeing with you in the abstract that a theoretically strong enough AI could identify bad papers, especially if it had some help for 'real'…

> Detecting claims that are themselves implausible seems like the hardest task possible, however. That's very close to general AI.

Yes, of course. I was simply trying to say that an AI can be quite successful in detecting the usual no-nos, eg. multiple comparisons without correcting for it, p-hacking, or ... who knows what "feature" the classifier would find. Maybe there's simply none, so it'll be really up to subject matter experts to review them. (But it's unlikely, because there are quite successful blogs devoted to simply picking apart shoddy papers simply based on looking at the controls, and other parts of experiment design and the methods sections, and of course the aforementioned stats.)