seems like this is just reward hacking the llm as a judge. this does not give you a story humans will be more likely to read imho
AlphaWrite: AI that improves at writing by evolving its own stories
11–20 of 162 posts
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#12I can't imagine LLMs are good judges of good writing.
That's the core of my concern too. Be interested to see what happens if you feed the ranking algorithm a list of the most popular books and a list of the most impactful books. Something tells me this will be a lot more interested in Chuck Tingle than Kafka.
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#13This feels like a lot of fluff, without some solid examples of the results - the one example of generated prose that they do provide is pretty unimpressive, it reads like… well, like an LLM wrote it.
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#14Note: Not associated with Google Deepmind (AlphaFold, AlphaGo, AlphaEvolve, etc.)
Nor associated with AlphaSmart word processors or their AlphaWrite application. Sigh.
In this genre do you really expect a lot of concern for intellectual property or the ability to identify the source of anything?
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#15If there is something that I would like AI to never touch, it's that. Please stop making the world worse.
You cannot stop people from making the world worse or better. The best you can do is focus on your own life.
In time many will say we are lucky to live in a world with so much content, where anything you want to see or read can be spun up in an instant, without labor.
And though most will no longer make a living doing some of these content creation activities by hand and brain, you can still rejoice knowing that those who do it anyway are doing it purely for their love of the art, not for any kind of money. A human who writes or produces art for monetary reasons is only just as bad as AI.
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#16If there is something that I would like AI to never touch, it's that. Please stop making the world worse.
More capable AI systems make the world better. If you don't like AI written material on principle, you can simply choose not to read it. Follow human writers who don't use AI.
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#17I'm struggling a bit to understand the difference between the reported results in the blog post and the examples in the Github.
The blog states:
> "Alpha Writing demonstrates substantial improvements in story quality when evaluated through pairwise human preferences. Testing with Llama 3.1 8B revealed:
72% preference rate over initial story generations (95 % CI 63 % – 79 %) 62% preference rate over sequential-prompting baseline (95 % CI 53 % – 70 %) These results indicate that the evolutionary approach significantly outperforms both single-shot generation and traditional inference-time scaling methods for creative writing tasks."
But in all of the examples using Llama 3.1 8B on the Github that I could find, the stories with the top 5 highest final 'ELO' are all marked elsewhere as:
"generation_attempt": null
Where the 'variant' stories, which I take to be 'evolved' stories, are marked:
"generation_type": "variant", "parent_story_id": "897ccd25-4776-4077-a9e6-0da34abb32a4"
IE - none of the 'winning stories' have a parent story; they seem to have explicitly been the model's initial attempt. The examples seem to prove the opposite of the statement in the blog post.
Perhaps 'variants' are slightly outperforming initial stories on average (I don't have time to actually analyse the output data in the repo), though it seems unlikely based on how I've read it (I could be wrong!) and this might be borne out with far more iterations.
However, a really important part of creative writing as a task is that you (unfortunately) only get to tell a story once. The losing variants won't ultimately matter. So, if I've read it correctly, and all the winning stories are 'not evolved' - from the initial prompt - this is quite problematically different from the blog's claim that:
> "we demonstrate that creative output quality can be systematically improved through increased compute allocation"
Super interesting work - I'd love to be told that I'm reading this wrong! I was digging through in such detail to actually compare differently-performing stories line-for-line (which would also be nice to see - in the blog post, perhaps).
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#18Appropriate that the original title misspells "Writing": >AlphaWrite: Inference time compute Scaling for Writting
I found the entire first sentence nearly unreadable:
"Large languagenference time compute Scaling for Writing models have demonstrated remarkable improvements in performance through increased inference-time compute on quantitative reasoning tasks, particularly in mathematics and coding"
Am I just out of the loop on the current jargon, or is that indeed a terribly-written first sentence?
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#19If there is something that I would like AI to never touch, it's that. Please stop making the world worse.
You cannot stop people from making the world worse or better. The best you can do is focus on your own life. In time many will say we are lucky to live in a world with so much content, where anything you want to see or read can be spun up in an instant, without labor. And though most will no longer make a living doing some of these content creation activities by hand and brain, you can still rejoice knowing that thos…
> In time many will say we are lucky to live in a world with so much content, where anything you want to see or read can be spun up in an instant, without labor.
Man, you are talking about a world that's not just much worse but apocalyptically gone. In that world, there is no more art, full stop. The completeness and average-ness of stimulation would be the exact equivalent of sensory deprivation.
Re: AlphaWrite: AI that improves at writing by evolving its own stories
#20Why would we ever want to build something like this, unless your goal is to have fiction writers make even less money than they already do. Just stop, please. Try and automate some horrible and repetitive drudgery. Do you want to live in a world where humans no longer do any creative work? It’s grotesque.
I want to live in a world with more options and freedom to choose. If somebody wants to build and explore with these AI systems, you can't stop them.