This would be useful for RAG when a Wiki doesn't exist. findOrCreate
Storm: LLM system that researches a topic and generates full-length wiki article
71–80 of 104 posts
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#72Earlier quoted context omitted.
> On the one hand, a tool is as good or bad as the person wielding it. I think the real reason is one line dogmas like this.
I'm not sure I follow you - reason for what? To be clear - I'm with you that these systems can absolutely be a force for vast good (at least, I think that was what you were getting at unless there was a missing '/s'). I use them daily to pretty astounding effect. I'll admit to being a little put off by being labeled dogmatic - it's not something I consider myself to be.
it was a half sentence, for that I apologize. and I don't remember entirely what I meant.
However, I do see a lot of one-sentence "truthms" being thrown around. like "garbage in; garbage out" and the likest.
these are not correct. we can just look at the current state of the art with LLMs that has vast amounts of garbage going in - it seems like the value is in the vastness of the data over the quality.
> On the one hand, a tool is as good or bad as the person wielding it.
I see this as being a dogme. smart people make good LLMs dumb people do not. but this is an open question. it seems like the biggest wallet will be the winner of the LLM game.
please correct me if I misunderstood something.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#73Small thing, but the blurb on the README says
> While the system cannot produce publication-ready articles that often require a significant number of edits, experienced Wikipedia editors have found it helpful in their pre-writing stage.
So it can't produce articles that require many edits? Meaning it can produce publication-ready articles that don't need lots of edits? Or it can't produce publication-ready articles, and the articles produced require lots of edits? I can't make sense of this statement.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#74I can see this being useful iif the content is generated on demand and then discarded. Publishing AI generated material is generally speaking a horrible idea and does nobody any good (at least until accuracy levels get much much better.) Even if they do it well and truthfully (which they don't) current LLMs can only summarize, digest, and restate. There is no non-transient value add. LLMs may have a place to help que…
are llms able to look at a list of categories, read content and then determine which of the categories apply?
Absolutely
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#75Small thing, but the blurb on the README says > While the system cannot produce publication-ready articles that often require a significant number of edits, experienced Wikipedia editors have found it helpful in their pre-writing stage. So it can't produce articles that require many edits? Meaning it can produce publication-ready articles that don't need lots of edits? Or it can't produce publication-ready articles,…
It gives you a draft that you should keep working on. For example, fact checking.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#76Small thing, but the blurb on the README says > While the system cannot produce publication-ready articles that often require a significant number of edits, experienced Wikipedia editors have found it helpful in their pre-writing stage. So it can't produce articles that require many edits? Meaning it can produce publication-ready articles that don't need lots of edits? Or it can't produce publication-ready articles,…
[deleted]
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#77I can see this being useful iif the content is generated on demand and then discarded. Publishing AI generated material is generally speaking a horrible idea and does nobody any good (at least until accuracy levels get much much better.) Even if they do it well and truthfully (which they don't) current LLMs can only summarize, digest, and restate. There is no non-transient value add. LLMs may have a place to help que…
are llms able to look at a list of categories, read content and then determine which of the categories apply?
This could be achieved by generating embeddings of suitable representations of the categories once, and then embedding the content at runtime, before using some distance metric to find matching categories for the content embedding.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#78I can see this being useful iif the content is generated on demand and then discarded. Publishing AI generated material is generally speaking a horrible idea and does nobody any good (at least until accuracy levels get much much better.) Even if they do it well and truthfully (which they don't) current LLMs can only summarize, digest, and restate. There is no non-transient value add. LLMs may have a place to help que…
This is categorically untrue. Publishing material generated like this is going to be generally better than human generated content. It takes less time, can be systematically tested and rigorous, and you can specifically avoid the pitfalls of bias and prejudice.
A system like this is multilayered, with prompts going through the whole problem solving process, considering the information presented, assuring quality and factuality, assigning the necessary citations and documentation for claims.
Accuracy isn't a problem. The way in which AI is used creates the problem - ChatGPT and most chat based models are single pass, query/response type interactions with models. Sometimes you get a second pass with a moderation system, doing a review to ensure offensive or illegal things get filtered out. Without any additional testing and prompt engineering, you're going to run into hallucinations, inefficient formulations, random "technically correct but not very useful" generations, and so forth. Raw ChatGPT content shouldn't be published without significant editing and going through the same quality review process any human written text should go through.
What Storm accomplishes is an algorithmic and methodical series of problem solving steps, each of which can be tested and verified and validated. This is synthesized in a particular way, intended as a factual reference article. Presumably you could insert debiasing and checks for narrative or political statements, ensuring attribution and citation occur for quotations, and rephrasing anything generated by the AI as a neutral, academic statement of fact with no stylistic and artistic features.
This is significantly different from the almost superficial interactions you get with chatbots, unless you specifically engineer your prompts and cycle through similar problem solving methods.
Tasks like this are well within the value add domain of current AI capabilities.
Compared to the absolute trash of SEO optimized blog posts, the agenda driven, ulterior laden rants and rambles in social media, and the "I'm oh-so-cleverly influencing the narrative" articles posted to Wikipedia by humans, content like this is a clear winner in quality, in my opinion.
AI isn't at the point where it's going to spit out well grounded novel answers to things like "what's the cure for cancer?" but it can absolutely produce a principled and legible explanation of a phenomenon or collection of facts about a thing.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#79From my experiments, this thing is pretty bad. It mixes up things that have similar names, it pulls in entirely unrelated concepts, the articles it generates are mind-numbingly repetitive and verbose (although notably with slightly different "facts" each time things are restated), its citations are often completely unrelated to the topic at hand, and facts are cited by references that don't back them up.
I mean, the spelling and syntax of the sentences is mostly correct, just like any LLM content. But there's ultimately still no coherence to the output.
Re: Storm: LLM system that researches a topic and generates full-length wiki article
#80Oh dear lord .... sub heading states - Storm - Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models Good luck with this storm, wiki's the world over. Just a thought but ... maybe someone should ask an org like the Internet Archive to snap-shot Wikipedia asap and label it Pre-Storm and After-Storm
LLM mediocrity is just a reflection of human mediocrity, and my bet is on the average LLM to get way better much faster than the average human doing the same.
Humans are limited in the volume of garbage they can produce.