Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is using AI tooling for a PhD literature review dishonest?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#21

Post by latand6 »

austinjp wrote: post #19
latand6 wrote: post #14

Earlier quoted context omitted.

Can you really reproduce it though? I thought it’s the experiments that have to be able to reproduce, not the literature review

Whether you can or can't in reality is moot, unfortunately. The literature search in biomedical fields should indeed be theoretically reproducible. I don't know about other fields, but it would seem odd to me if a search was not reproducible, that would lead to a very arbitrary literature selection. As for the experiments, yes, in experimental fields. But in all (most?) fields, including non-experimental, the whole p…

Got that, thanks for the advice, I'll ask my supervisor how to address that properly

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#22

Post by Acacian »

The verification pipeline is the most valuable part of your workflow. Most people who use AI for literature reviews skip exactly that step — they trust the output and move on.

What you're describing is closer to building a testing harness than "using AI to write." You're asserting claims, checking them against source PDFs, and reviewing manually. That's more rigorous than most manual lit reviews where people skim abstracts and cite papers they half-read.

Document the pipeline as methodology in your dissertation. That turns a potential misconduct question into a contribution.

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#23

Post by latand6 »

Acacian wrote: post #22

The verification pipeline is the most valuable part of your workflow. Most people who use AI for literature reviews skip exactly that step — they trust the output and move on. What you're describing is closer to building a testing harness than "using AI to write." You're asserting claims, checking them against source PDFs, and reviewing manually. That's more rigorous than most manual lit reviews where people skim abs…

Thanks for the feedback and appreciation. Maybe at some point I’ll polish it and make it open source if people find this valuable

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#25

Post by gsch1 »

latand6 wrote: post #14
austinjp wrote: post #10

While your dashboard sounds fancy, this part raises issues: > I run ChatGPT Pro to collect all relevant papers Any literature review must be reproducible. If you can't say exactly what queries you ran against exactly what databases, you'll get into trouble. Whether or not that's the way things should be is irrelevant: it's the way things are. You should ask your supervisor if your approach is okay. If necessary, ask…

Can you really reproduce it though? I thought it’s the experiments that have to be able to reproduce, not the literature review

Yes, you must be able to reproduce the results. If reproduction is not possible, the work lacks scientific validity.

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#26

Post by BrenBarn »

austinjp wrote: post #18
BrenBarn wrote: post #11

Earlier quoted context omitted.

> Any literature review must be reproducible. That's totally at odds with my understanding, but perhaps this differs between fields.

Quite probably there are differences between fields. In biomedical literature reviews the search terms and databases are detailed, and (in systematic reviews) a PRISMA flowchart [0] provided. The theory being that other researchers could repeat the searches and the in/out decisions and get the same stack of papers to review. [0] https://www.prisma-statement.org/prisma-2020-flow-diagram

Okay yeah that sounds closer to what I'd call a meta-analysis. In linguistics (which is the field I was in) "literature review" just means "someone looked around and read some papers they thought might be related". There's no expectation that it will be systematic in any replicable way.

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#27

Post by psyklic »

Everyone dislikes pedantically verifying references. However, if you cut corners here then will you also cut corners pedantically verifying research results?

Beyond references, the point of the literature review is to ensure you have read the literature and understand it well enough to accurately summarize it. If you present a literature review, it's likely assumed you did all of this. So at the very least you should be upfront about how an LLM assisted you.

Re: Ask HN: Is using AI tooling for a PhD literature review dishonest?

#28

Post by ajdecon »

> The idea of the dashboard is the following: I run the Codex through a web chat to identify the relevant quotes — relevant for my dissertation topic — and how they are relevant, it combines them into a number of claims connected with each quote with a link. And then I review each quote and each claim manually and tick the boxes.

I’d be most concerned about this component of your process, tbh. IIUC, you’re not just using the LLM to identify relevant papers, i.e. a fancy search engine. You’re also extracting specific statements, divorced from their context in a given paper, and using these to make claims for your research.

Even if you validate that the quotes are actually present in the papers, are you also reading the full papers to ensure you understand the overall results of the paper and what the quotes mean in that context? Or are you just identifying hopefully-relevant snippets and combining them?

Post reply on HN