Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Storm: LLM system that researches a topic and generates full-length wiki article

github.com

Post reply on HN
Print view View original

21–30 of 104 posts

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#21

Post by howenterprisey »

tossandthrow wrote: post #16
poyu wrote: post #9

Earlier quoted context omitted.

> Ai is not able to create original knowledge The current state of LLMs do hallucinate though. It's just not a very trustworthy source of facts.

just like my first teachers said I should absolutely not use Wikipedia. LLMs was popularized less than 2 years ago. I think it is safe to assume that it will be as trustworthy as you see Wikipedia today, and probably even more as you can embed reasoning techniques into the LLMs to correct misunderstandings. Wikipedia cannot self correct.

Wikipedia absolutely self-corrects, that's the whole point!

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#22

Post by Terr_ »

madeofpalk wrote: post #17
whitehexagon wrote: post #11

Hmm something about this title containing the word 'research' disturbs me. I associate that word with rigorous scientific methods that leads to fact based knowledge or maybe some new hypothesis, not some LLM hallucinating sources, references, quotes and all the other garbage they spit out when challenged over a point of fact. Horrifying to think peeps might turn towards these tools for factual information.

This anthropomorphism really bothers me. These tools are useful for what they’re good for, but I really dislike the agency people keep trying to give to them.

I think there's always been fine line between anthropomorphism as a metaphorical way to indicate complexity versus a pitfall where people (especially outside of a field) start acting like it's a literal statement.

Ex: "the gyroscope is trying to stay upright", or "the computer complains because the update is broken" or "evolution will give the birds longer beaks".

That said, I agree that the problem is dramatically more-severe when it comes to "AI".

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#23

Post by pksebben »

barbarr wrote: post #7

I guess this is a good thing for increasing coverage of neglected areas. But given how cleverly LLMs can hide hallucinations, I feel like at least a few different auditor bots should also sign off on edits to ensure everything is correct.

This method has actually been proven effective at increasing reliability / decreasing hallucinations [1]

1 - https://arxiv.org/abs/2402.05120

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#25

Post by pksebben »

LeoPanthera wrote: post #4

I saved a full snapshot of Wikipedia (and Stack Overflow) in the weeks before ChatGPT launched, and every day I'm more glad that I did. They will become the Low Background Steel of text.

That's gonna be a lot of fun to play with in a year or so.

There's a concurrent explosion of 'veracity' analysis - it'll be fun to run those against wikipedia a year from now and your data.

Incidentally, are you interested in mirroring your dataset and making it more robust? I'm sure I've got a few TB of storage lying around somewhere...

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#27

Post by lukev »

I can see this being useful iif the content is generated on demand and then discarded.

Publishing AI generated material is generally speaking a horrible idea and does nobody any good (at least until accuracy levels get much much better.)

Even if they do it well and truthfully (which they don't) current LLMs can only summarize, digest, and restate. There is no non-transient value add. LLMs may have a place to help query, but there is no reason to publish LLM regurgitations alongside the ground truth used to generate them.

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#28

Post by tiptup300 »

lukev wrote: post #27

I can see this being useful iif the content is generated on demand and then discarded. Publishing AI generated material is generally speaking a horrible idea and does nobody any good (at least until accuracy levels get much much better.) Even if they do it well and truthfully (which they don't) current LLMs can only summarize, digest, and restate. There is no non-transient value add. LLMs may have a place to help que…

are llms able to look at a list of categories, read content and then determine which of the categories apply?

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#29

Post by Anon84 »

pksebben wrote: post #25
LeoPanthera wrote: post #4

I saved a full snapshot of Wikipedia (and Stack Overflow) in the weeks before ChatGPT launched, and every day I'm more glad that I did. They will become the Low Background Steel of text.

That's gonna be a lot of fun to play with in a year or so. There's a concurrent explosion of 'veracity' analysis - it'll be fun to run those against wikipedia a year from now and your data. Incidentally, are you interested in mirroring your dataset and making it more robust? I'm sure I've got a few TB of storage lying around somewhere...

You can just download it yourself. Wikimedia publishes regular dumps in easily accessible formats: https://dumps.wikimedia.org/enwiki/20240320/ (the most recent for english Wikipedia)

Re: Storm: LLM system that researches a topic and generates full-length wiki article

#30

Post by WhitneyLand »

jakderrida wrote: post #8
LeoPanthera wrote: post #4

I saved a full snapshot of Wikipedia (and Stack Overflow) in the weeks before ChatGPT launched, and every day I'm more glad that I did. They will become the Low Background Steel of text.

The thing is that the Wiki mods will need to be more diligent with uncited things. I also see 2 massive opportunities here. First is that they can have agents check the cited source and verify whether the source backs up what's said to a reasonable degree. Second opportunity is fitting in things only found in other language Wikis that either be incorporated into the english one or help introduce new articles. Believe…

>>LLMs can't generate english answers for things answered only in Russian in the training data.

For multilingual LLM’s? Why do you think that?

An LLM can translate inputs of arbitrary Russian text. If there were an English question about something only in the training data as Russian, I would expect an answer - with the quality being on par with its general translation capabilities.

Post reply on HN