Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TLDR: Extreme Summarization of Scientific Documents

arxiv.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: TLDR: Extreme Summarization of Scientific Documents

#31

Post by msamwald »

etaioinshrdlu wrote: post #10

I played around with this on the demo page and found that while the generated "TLDR" are pretty good, they tend to generate sentences composed of fragments of existing sentences. Basically, it seems vaguely extractive in nature. Never did I see it summarize a concept in new words, or try to dumb down a complicated concept further than the original paper did. Given the results of GPT3 I would think that it should be p…

Being very efficient at mostly extractive summarization and abstaining from abstractive summarization does seem a better bet though, because fewer things can go wrong and it is easier to check the summaries against the full text.

Re: TLDR: Extreme Summarization of Scientific Documents

#32

Post by austinjp »

justarandomq wrote: post #7

What you get from applying TLDR to their paper: We introduce SCITLDR, a new multi-target data set of 5.4KTLDRs over 3.2Kpapers. Keeping pdf's copy-paste artifacts: We introduceTLDRgeneration, a new formof extreme extreme summarization, for scientific pa-pers. Adding intro and conclusion (optional): We introduce SCITLDR, a new data set of 5.4KTLDRs over 3.2Kpapers. [0] https://scitldr.apps.allenai.org/

Yeah. I happen to have been looking at this problem in my spare time recently. I tried a bunch of abstractive AIs and approaches, and none produce consistently usable results.

I'm sticking with extractive approaches plus a bunch of hard-coded general and domain-specific rules for now.

Re: TLDR: Extreme Summarization of Scientific Documents

#34

Post by FL33TW00D »

arolihas wrote: post #16
etaioinshrdlu wrote: post #10

I played around with this on the demo page and found that while the generated "TLDR" are pretty good, they tend to generate sentences composed of fragments of existing sentences. Basically, it seems vaguely extractive in nature. Never did I see it summarize a concept in new words, or try to dumb down a complicated concept further than the original paper did. Given the results of GPT3 I would think that it should be p…

I think you’re underestimating how hard what you’re describing is. GPT-3 can mimic the language of reasoning but that doesn’t mean it’s capable of higher order reasoning.

After reading through https://www.gwern.net/GPT-3 , I suspect that GPT-3 is capable of higher order reasoning, given the right motivation (prompt).

Re: TLDR: Extreme Summarization of Scientific Documents

#36

Post by qayxc »

pixiemaster wrote: post #23

English only, as usual for NLG

In this particular case it's excusable as English is the Lingua Franca of science today. Used to be Latin, then French and German, now it's English. No big deal IMO.

In fact I kind of like the way this is going since it represents a fantastic opportunity for NL researchers to stand out simply by publishing research and corpora focused exclusively on low-resource languages and non-English/Mandarin in general.

It is also important to note that most of the ML research in the field is pretty much language agnostic and is concerned with general concept such as efficient en-/decoding [1], training methods [2], and even stealing pre-trained weights from APIs (like GPT-2 or even 3) without paying for training [3] :)

It's just easier to get your hands on and verify English corpora, results and pre-trained models for reproducibility than say Mongolian or Gaelic so that's a factor, too.

[1] https://arxiv.org/pdf/1904.09751.pdf

[2] https://arxiv.org/pdf/2003.10555.pdf

[3] https://arxiv.org/pdf/1910.12366.pdf

Re: TLDR: Extreme Summarization of Scientific Documents

#37

Post by KineticLensman »

ultrastito wrote: post #12

That's what abstract are for

In some situations abstracts serve as bibliographic metadata rather than a summary of the content. Examples includes cases where the content is hidden behind a paywall or, in defence, when a paper's content is classified in some way but the existence of the paper itself is not. In both cases, the abstract may help you decide whether it is worth accessing the full paper, but on its own won't give you an answer. E.g "we studied X" but not "and concluded Y".

Obviously abstracts can include a content summary as well as bibliographic metadata, but not all do.

Re: TLDR: Extreme Summarization of Scientific Documents

#38

Post by kreeben »

justarandomq wrote: post #21
ultrastito wrote: post #12

That's what abstract are for

After some thought I agree with you that this is the wrong problem to solve. I took a narrative detour I wanted to share: Suppose we make the analogue of a scientific paper to a piece of mineral ore (in terms of their raw content, and without written symbols in them for the sake of the analogy) extracted from some mine or quarry. This ore is somehow useful to someone, even if its value is structural: the shingles on…

>> this is the wrong problem to solve

Highly intelligent human beings have a natural ability to summarize big ideas into TLDRs. Are humans basically a bunch of "summarizers"? Probably not. Is this ability to summarize or compress big ideas into smaller, more condensed pieces of information, important to the human race? Yes, I would say that they are. So to me, this is certainly one of those problems that we correctly attempt to solve.

Re: TLDR: Extreme Summarization of Scientific Documents

#39

Post by capableweb »

ultrastito wrote: post #12

That's what abstract are for

Speaking of abstracts, did you read the abstract from the paper you're commenting about? I don't think you did, because it outlines how this approach is different, and maybe if there was something better than the abstract, you would have read it and not assume it's the same as what we're using abstracts for today.

In short, here's the major differences:

> SciTLDR contains both author-written and expert-derived TLDRs

> CATTS improves upon strong baselines under both automated metrics and human evaluations

Re: TLDR: Extreme Summarization of Scientific Documents

#40

Post by arolihas »

FL33TW00D wrote: post #34
arolihas wrote: post #16

Earlier quoted context omitted.

I think you’re underestimating how hard what you’re describing is. GPT-3 can mimic the language of reasoning but that doesn’t mean it’s capable of higher order reasoning.

After reading through https://www.gwern.net/GPT-3 , I suspect that GPT-3 is capable of higher order reasoning, given the right motivation (prompt).

It’s impressive but doesn’t the need for a “good” prompt kind of show it doesn’t have the strong reasoning required to do the task you’re describing? Also there’s some interesting critique here https://www.lesswrong.com/posts/ZHrpjDc3CepSeeBuE/gpt-3-a-di...
Post reply on HN