Live data from Hacker News

AI vs. Professional Authors Results

mark---lawrence.blogspot.com

81–90 of 103 posts

Re: AI vs. Professional Authors Results

#81

I think if you compared the AI stories to works by “top” authors, the results wouldn’t really be as close. No one is confusing a story by Kafka or Conrad with a ChatGPT one. Because unfortunately, one reason why readers can’t tell the difference between the AI and human authors is because they don’t have much exposure to the greats. The average person reads something like 2 books a year, and they probably aren't read…

That's probably true, but as the author points out, it's still interesting to see where the boundary is at the moment. It's a lot further along than people typically argue imo.

I don’t really find it that surprising - if you write generic low-quality stories, it’s difficult to differentiate your work from an AI writing generic low-quality stories. People have been selling slop stories on Amazon for a long time before AI tools, so describing them as professional authors is not exactly damning here.

Re: AI vs. Professional Authors Results

#82

I think if you compared the AI stories to works by “top” authors, the results wouldn’t really be as close. No one is confusing a story by Kafka or Conrad with a ChatGPT one. Because unfortunately, one reason why readers can’t tell the difference between the AI and human authors is because they don’t have much exposure to the greats. The average person reads something like 2 books a year, and they probably aren't read…

With all due respect but at least Robin Hobb, is one of the greats in her domain.

Re: AI vs. Professional Authors Results

#83

Earlier quoted context omitted.

> it just generates the same old fairy tale plots using the new words it has learned. I think you're leaving out the best part! I don't want to spoil it, it's a short story. Classic trope, but still. Story here[0] On another note, as an avid SciFi lover I have always found it interesting that in books, movies, and shows there have been many machines that talk and do complex tasks yet no one ever thought they were ali…

Because it's fiction and the Author is God. In Star Trek, the computer is framed as an appliance. It's the ship's operating system. The characters treat it like a highly advanced Alexa. They issue commands ("Tea, Earl Grey, hot"), ask for information, and expect a transactional response. No one ever asks the computer, "How are you feeling today?" because the narrative has established it doesn't have feelings. It's a…

  > Because it's fiction and the Author is God.

  >> These are fiction, but I find this so interesting
I mean... I do recognize this fact. I hope we're clear on that.

  > The characters treat it like a highly advanced Alexa.
I see a lot of people use GPT the same way.

But also, I disagree. People do ask "How are you feeling today?" to the holo programs. Hell, Paris makes a joke to Kim about how everyone falls in love with a holo character at some point. That it is the fantasy.

  > 'The Author' could have easily positioned the computer or the holodeck in a similar manner and people would agree it was sentient.
I mentioned [1] for a good reason. There were more than one episode addressing this point. Not to mention the entire Voyager where this is a subplot of the entire series.

  > Or Star Wars droids could easily be given more of this kind of weight than they are currently given.
I disagree. Some feel very alive.

I get your point and there's a lot I agree with it but I think you're brushing things off too quickly. You can't just say that people have no free interpretation and "the author" fooled everyone. Especially where there are plenty of stories and episodes which bring all this into question. Please, go read [0]

Re: AI vs. Professional Authors Results

#84
post #27

Earlier quoted context omitted.

I would love to read some of this. Where do you find AI generated fiction?

https://www.royalroad.com/fictions/search?globalFilters=fals... Though I’ll admit I can’t speak to the quality of that except my own stuff (which I’m naturally predisposed to like). This was my attempt at fully AI generated (though edited by human): https://www.royalroad.com/fiction/101072/inherited-wounds

I've had two separate experiences reading a story on RoyalRoad, getting ten chapters in, noticing that any individual chapter is technically great but I just don't care about the characters or plot can't be bothered reading further, and then noticing the story was AI-assisted.

I think the AI seems to struggle with consistency of characters and themes, and particularly with character growth over time: it can write touching moments, but these don't fit properly with the character's actions before and then after. It reads a bit like a story written by a hundred professional authors who can skim-read all the previous chapters but are on a strict time limit and don't have access to each other's notes. This makes me wonder if they're just not giving the AI notes on structure and character.

Re: AI vs. Professional Authors Results

#85

Earlier quoted context omitted.

I don't know if anyone has officially said so but there was a public statement about it being chosen as a reference to story telling (Celtic language). So could be for similar reasons or could be a reference. Not surprising considering how famous the story is and how famous Asimov is. But maybe someone else knows more definitively.

Celtic? The name "Gemini" is much, much, much better known from Greek myth, if it occurs in Celtic culture at all...

Here, I'll rephrase thenoblesunfish for clarity, because you seem to have misread

  > Is this why the Google product (now just called Gemini) was called [Bard]?
"That" == "Bard".

They weren't referring to Gemini, which is why there's that whole thing in parentheses stating it's *now* called Gemini

From Wiki

  | The technology was developed under the codename "Atlas", with the name "Bard" in reference to the Celtic term for a storyteller and chosen to "reflect the creative nature of the algorithm underneath".

Re: AI vs. Professional Authors Results

#86

Earlier quoted context omitted.

Because it's fiction and the Author is God. In Star Trek, the computer is framed as an appliance. It's the ship's operating system. The characters treat it like a highly advanced Alexa. They issue commands ("Tea, Earl Grey, hot"), ask for information, and expect a transactional response. No one ever asks the computer, "How are you feeling today?" because the narrative has established it doesn't have feelings. It's a…

> Because it's fiction and the Author is God. >> These are fiction, but I find this so interesting I mean... I do recognize this fact. I hope we're clear on that. > The characters treat it like a highly advanced Alexa. I see a lot of people use GPT the same way. But also, I disagree. People do ask "How are you feeling today?" to the holo programs. Hell, Paris makes a joke to Kim about how everyone falls in love with…

>I see a lot of people use GPT the same way.

Some people do, but not everyone, because LLMs are capable of being more than that. The problem with the fictional setting is that this transactional use is often all you see, because that's the way the author has chosen to frame the story.

In real life, even a person who primarily uses an LLM as a tool may conclude it's intelligent after a particular conversation. Because if the LLM is capable of more than being an advanced Alexa, then at least you can discover that through your own personal use.

In fiction? You're stuck with whatever the author wants to focus on. How do you know if the Enterprise computer can be more than an advanced Alexa? It's not like you can use it yourself. You only know what the author shows you.

Your point about Voyager and The Doctor doesn't detract from mine; it's a good example of it. The computer like entity isn't something that's we are supposed to treat with potential sentience, until the Author decides that it is.

>But also, I disagree. People do ask "How are you feeling today?" to the holo programs. Hell, Paris makes a joke to Kim about how everyone falls in love with a holo character at some point. That it is the fantasy.

I was talking about the main computer, but regardless, don't you see how the framing is still there even with the holograms? As you said, Paris makes a joke about it. It's treated as a silly phase, something unserious that people grow out of. The narrative is telling the audience not to take it too seriously.

>> Or Star Wars droids could easily be given more of this kind of weight than they are currently given. I disagree. Some feel very alive.

And that's exactly the point. They feel very alive, yet how many people (in the audience or in the story) are concerned that these seemingly sentient beings are treated as slaves and second-class citizens? Very few. Why? Because 'The Author' is not interested in telling that story. Characters only have as much fidelity as the Author wants.

>You can't just say that people have no free interpretation and "the author" fooled everyone.

It's not about lacking interpretation or being "fooled." It's the simple fact that a story is biased at its core by the author's focus and intent. You are seeing the world through the Authors lense. You can only form an interpretation on what the author has provided.

Very different from being able to spin up ChatGPT or Gemini or whatever and form your own conclusions from your own personal usage.

Re: AI vs. Professional Authors Results

#87

Earlier quoted context omitted.

> Because it's fiction and the Author is God. >> These are fiction, but I find this so interesting I mean... I do recognize this fact. I hope we're clear on that. > The characters treat it like a highly advanced Alexa. I see a lot of people use GPT the same way. But also, I disagree. People do ask "How are you feeling today?" to the holo programs. Hell, Paris makes a joke to Kim about how everyone falls in love with…

>I see a lot of people use GPT the same way. Some people do, but not everyone, because LLMs are capable of being more than that. The problem with the fictional setting is that this transactional use is often all you see, because that's the way the author has chosen to frame the story. In real life, even a person who primarily uses an LLM as a tool may conclude it's intelligent after a particular conversation. Because…

I still think you're cherry-picking and ignoring any of the depth here. You're being so quick to find an answer you are missing all complexity. Your argument is that people are only subjected to what the author says, leading them to have no ability to think or form conclusions themselves. Star Trek was given as the most familiar example (one which people would likely debate these aspects) but there is far more complexity in stories like The Positronic Man or even A.I., where the author is specifically asking you to think about these things (just like in those aforementioned Trek episodes where clearly the author is making people think about those things). I'm just trying to say, don't let an answer get in the way of understanding. Don't just trivialize everything, and don't try to explain to someone what they already acknowledged.

Re: AI vs. Professional Authors Results

#88

The story plot was "Meeting a dragon". As both a human and a writer, challenge accepted: Long ago, there lived a golden dragon whose fractal-like scales gleamed in the glow of the morning in her cave. She was known for her kindness, and many came not with sword or spear, but with humble requests - for you see, it was widely believed that the mystical scales of a dragon would heal illness, cure ailments, and provide f…

I think that humans do executive decision making better than LLMs.

So, yeah, your "Meeting a dragon" story was about a single point - an attempt at a humourous twist ending; you built your story around that.

My approach would be having the Dragon in the title be a metaphor, for something powerful, dangerous and scary:

1. Comedy: new g/friend meeting MiL (the Dragon) for the first time

2. Thriller: guy finds out what his wife (The Dragon) really is like, basically the plot to Gone Girl

3. Drama: An alcoholic anthopormorphising the addiction (The Dragon) as an uncontrollable beast within himself

4. Historical: An author examining the events of the Tuskagee Syphilis experiment in the legislated-racism (The Dragon) period of the time.

5. SciFi: "The Dragon is a Harsh Mistress" (enough said)

6. Action: The dragon is a legendary elder of a mystical martial arts sect ("Enter the Dragon" stuff)

7. Fantasy: An actual, literal Dragon!

This is before I've considered a single line of plot, a single character, or character motivation. It's before I considered tone and presentation (Narrative? First-person narration? Dialog-driven narration?)

I mean, before getting to actual plot, characters, setting, tone, etc ... I've already got my message figured out. That's executive decision making.

The LLM will not, when given the directive "Write the story 'Meeting the Dragon'" perform any executive decision making. You have to baby it through every step. Basically micromanage it.

Re: AI vs. Professional Authors Results

#89

Here are my notes and guesses on the stories in case people here find it interesting. Like some others in the blog post comments I got 6/8 right: 1.) probably human, low on style but a solid twist (CORRECT) 2.) interesting imagery but some continuity issues, maybe AI (INCORRECT) 3.) more a scene than a story, highly confident is AI given style (CORRECT) 4.) style could go either way, maybe human given some successful…

I was surprised at the result, and even more surprised when I read that one of the authors who did the test got 4 out of 5 wrong, and rated 2 of the AI stories highly.

Looking at my notes, I got one wrong (story 5, dunno what the "name" was supposed to be, assumed that the "name" is something widely-known in culture that brings about the end times, a something that I didn't know about, and so marked it as Human because of a supposed reference to a shared cultural knowledge), and all the AI written stories I rated at either 1 or two points, with the lowest Human-written story getting 3 and the highest getting 5 (Story 1).

It makes me wonder if we are over-estimating the skill an author has when reading based on their demonstrated skill when writing.

IOW, according to my notes/performance, the AI stories were easy to spot and correlated with low scores anyway, while the author(s), who actually produced high-rated stuff for me, rated my low-rated stuff as high.

Re: AI vs. Professional Authors Results

#90
I'm very surprised according to results people struggled with identifying [3] and [4] as AI.

IMO both are simply bad and both contain usual telltales in spades (continuity problems, failed or trite metaphors/analogies, semantic failures, overall feeling of 'wtf is even being attempted here').

I'm not so surprised that people struggled with identifying [1] as human - the confounding factor is that this flash story is unpleasantly written, and it's not easy to realize that its failure modes (eg. trying to cram too much in too short a text) are rather human like. And I'm sure the fact that arguably the hardest to digest and rather bad human story opens the poll might somewhat influence the further analyses.

As others in the poll I failed to identify [5] as AI even though in hindsight the telltales are also there. That's because I rather liked it, and as a result it was harder to be vigilant. I also was very undecided on [8]. Finally I scored 6/8, but I wouldn't say it was easy.

Shame that comparing to the previous contest https://mark---lawrence.blogspot.com/2023/09/so-is-ai-writin... is not straightforward. In that one I scored 9/10 while having very easy time (I didn't even finish reading some of them before making up my mind). I also felt completely excused with my only failure, incorrectly identifying as AI the story written in the style of exhaustingly banal fan fiction. But frankly I found almost all the human stories in the previous edition better then the current ones.

In retrospect ChatGPT4 was a terrible writer. ChatGPT5 seems to be an improvement to the admittedly worrying point. Still not impossible to discover though.

However these are my impressions only and it looks maybe I was lucky and I should not generalize it? According to the website people had serious trouble discerning gpt4 writing also 2 years ago. And I'm rather shocked they did. And that they scored some of those banal AI stories positively.

If it's not luck on my part, then maybe discerning AI writing is a skill very different from 'writing' or 'being deeply interested in literature', skills of people who usually frequent this blog?

Post reply on HN