Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I built "AI Wattpad" to eval LLMs on fiction

narrator.sh

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Show HN: I built "AI Wattpad" to eval LLMs on fiction

#31

Post by rbtprograms »

jauws wrote: post #27
rbtprograms wrote: post #26

Earlier quoted context omitted.

have you read any of the generated stories? if you can honestly tell me this is not complete drivel (even worse, wildly generic and poorly written) then i will consider giving real feedback but i would find that hard to believe.

I hope it's clear that the stories aren't being generated one-shot. I'm sure there are flaws that I haven't perfectly accounted for in the agent-loop, but because we randomize the models for each of the brainstorming -> writing -> memory parts, bad intermediate outputs will affect the final output as well. That's why unless we have above average models across all 3 stages, it might be worse than what you're used to.…

so you havent read any of them, very nice product you got there. i stand by my original statement.

Re: Show HN: I built "AI Wattpad" to eval LLMs on fiction

#32

Post by permenant »

It would be interesting to consider composite systems where human brainstorming feeds AI writing, as well as vice versa, to see what kind of engagement with AI writing people like the most. At least in my case, I find plot writing good fun, and actual writing slightly less good fun.

Re: Show HN: I built "AI Wattpad" to eval LLMs on fiction

#35

Post by jauws »

JoshPurtell wrote: post #34

This is super cool! Have you tried GEPA?

Thanks Josh! I tried GEPA previously back when it was still 1-shot generation. It actually ended up working really well for some models and horrible for others, so I decided to scrap for a more generic prompt instead to make the benchmark a bit more rigourous.

Re: Show HN: I built "AI Wattpad" to eval LLMs on fiction

#36

Post by jauws »

permenant wrote: post #32

It would be interesting to consider composite systems where human brainstorming feeds AI writing, as well as vice versa, to see what kind of engagement with AI writing people like the most. At least in my case, I find plot writing good fun, and actual writing slightly less good fun.

Definitely on the to-do list! Right now, there's smth called fork (inspired by Github fork), where it lets you remix the story with a given input. It might be cool for you to mess around with.
Post reply on HN