Consensus: Use AI to find insights in research papers • Hacker News Board

Re: Consensus: Use AI to find insights in research papers

#11

Post by dvt » Thu, Aug 17, 2023, 10:19 PM UTC

input_sh wrote: post #10

I felt really excited clicking on some of the suggested prompts, but that excitement quickly fell apart when trying to generate a summary for what I consider to be a fairly simple custom search: impact of airbnb listings on house prices → can't summarise, need to post it as a question. how do airbnb listings impact house prices? → can summarise, can't create a concensus, must use a yes/no question. do more airbnb lis…

> And I gave up and deleted my account.

Same. Yet another "AI product" that has zero product-market fit and zero usefulness. What sucks is that even if you do get a consensus, it's not even tractable (e.g. does not properly cite sources), so where could I possibly even use the conclusion drawn?

Re: Consensus: Use AI to find insights in research papers

#12

Post by xigency » Thu, Aug 17, 2023, 10:29 PM UTC

cj wrote: post #9

fzliu wrote: post #8

Earlier quoted context omitted.

I'll play DA here - there's quite a bit of engineering surrounding these apps that can appear hidden to folks from the outside looking in. Various levels of prompt engineering and in-context learning might be necessary to get optimal results, and this could mean significantly more complexity at the application level.

Every time I hear or read "prompt engineering", I can't help but cringe a bit. I'm not sure why, but it's the same reaction I would have if I heard someone say "Google search query engineering". Comparing to google search, there definitely is skill involved in knowing how to google things well. We're all accustomed to googling things many times per day so I think a lot of people forget that being able to google thing…

Effective prompting has more to do with theater than engineering.

Re: Consensus: Use AI to find insights in research papers

#13

Post by [deleted] » Thu, Aug 17, 2023, 10:31 PM UTC

dvt wrote: post #11

input_sh wrote: post #10

I felt really excited clicking on some of the suggested prompts, but that excitement quickly fell apart when trying to generate a summary for what I consider to be a fairly simple custom search: impact of airbnb listings on house prices → can't summarise, need to post it as a question. how do airbnb listings impact house prices? → can summarise, can't create a concensus, must use a yes/no question. do more airbnb lis…

> And I gave up and deleted my account. Same. Yet another "AI product" that has zero product-market fit and zero usefulness. What sucks is that even if you do get a consensus, it's not even tractable (e.g. does not properly cite sources), so where could I possibly even use the conclusion drawn?

[deleted]

Re: Consensus: Use AI to find insights in research papers

#14

Post by SamBam » Thu, Aug 17, 2023, 10:32 PM UTC

didibus wrote: post #2

I've been doing this with chatGPT directly. But it's cutoff data is 2021. It's pretty good though at telling you even the research name so you can validate.

As someone who is starting a master's in science and will be looking at lots of research papers, I've been wondering what the best use of this could be.

If I have my own PDFs, I guess I could get ChatGPT to create summaries in some structured way, perhaps in a single file with citation:summary, and then send up that file with every question I ask?

Re: Consensus: Use AI to find insights in research papers

#15

Post by [deleted] » Thu, Aug 17, 2023, 10:41 PM UTC

input_sh wrote: post #10

I felt really excited clicking on some of the suggested prompts, but that excitement quickly fell apart when trying to generate a summary for what I consider to be a fairly simple custom search: impact of airbnb listings on house prices → can't summarise, need to post it as a question. how do airbnb listings impact house prices? → can summarise, can't create a concensus, must use a yes/no question. do more airbnb lis…

[deleted]

Re: Consensus: Use AI to find insights in research papers

#16

Post by pmarreck » Thu, Aug 17, 2023, 10:46 PM UTC

I asked it about UBI and I guess there isn't enough data yet, because it called it "fiscally unbearable and morally unacceptable" (which seems to be what only 1 uncited source said)! At least it admitted consensus was low...

https://consensus.app/results/?q=Does%20universal%20basic%20...

Summary Top 10 papers analyzed Some studies suggest that universal basic income (UBI) can generate support for structural reforms and improve mental health, while other studies argue it may be fiscally unbearable, morally unacceptable, and increase wealth inequality.

Re: Consensus: Use AI to find insights in research papers

#17

Post by permanent » Thu, Aug 17, 2023, 11:01 PM UTC

marcopicentini wrote: post #3

Why not reply directly with the answer learned from paper? Currently it’s like search the more relevant paper, open the pdf and read it.

Because providing always accurate LLM-based paraphrasing would be much more difficult.

Re: Consensus: Use AI to find insights in research papers

#18

Post by PNewling » Thu, Aug 17, 2023, 11:12 PM UTC

yuvalr1 wrote: post #7

> The current source material used in Consensus comes from the Semantic Scholar database For people who are familiar with this database, are the papers in it trustworthy? It's nice to search directly in scienific papers, but only if they're reputable.

I'm not super familiar with them, but taking a quick look it looks like the have a lot of papers published by the Allen Institute in Seattle[0] (which I am much more familiar with). So I would say they at least carry very reputable papers.

Here[1] is a summary of Semantic Scholar Database in the National Library of Medicine (note also from an Allen Institute spinoff, but it just made it easy to search for something I know as reputable).

[0] https://www.semanticscholar.org/author/Allen-Institute/21263... [1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5764585/

Re: Consensus: Use AI to find insights in research papers

#19

Post by orzig » Thu, Aug 17, 2023, 11:52 PM UTC

yuvalr1 wrote: post #7

> The current source material used in Consensus comes from the Semantic Scholar database For people who are familiar with this database, are the papers in it trustworthy? It's nice to search directly in scienific papers, but only if they're reputable.

Yes, I am a subscriber. They have little icons like “rigorous journal”, “systemic review”, “meta-analysis” and (if you give this any weight) “highly cited”. I believe you can filter by those in your search as well.

Re: Consensus: Use AI to find insights in research papers

#20

Post by orzig » Thu, Aug 17, 2023, 11:53 PM UTC

malux85 wrote: post #6

acqbu wrote: post #5

Is this another SaaS making API calls to OpenAI or is it more than that? If so, how?

Embed articles and throw the results in a vector database. Throw up a search result that just uses cosine similarity on the vector search with questionable metrics and no explanations on how things are calculated. Charge yearly because you know people will churn after a month or two. Profit - Every "AI startup" in the last 2 months

They also have a great Slack community and are rapidly turning out features. Everything I have seen suggests that they are competent and committed to the mission of making it easier to do good science.

It is easy to be cynical about the gold rush, but don’t throw the babies out with the bathwater.