Saturday 28 October 2023

A Literary Analysis of ChatGPT3.5

I've said plenty enough times throughout this countdown that robots cannot be creative, that they cannot replace the human mind when it comes to some creative tasks, such as drawing, acting and writing.
"But wait," I hear you say "A.I. can write. Heck, I can ask ChatGPT to write me a poem about 'a monster that's lonely because it killed everyone', and it will do it - look!"
In shadows deep, where moonlight fades,
A lonely monster, in solitude wades.
It's heart heavy with a dreadful cost,
For in its wake, all lives were lost.

A creature born of darkness and night,
It brought an end to the world's light.
But in its solitude, it weeps and moans,
A lonely monster with empty thrones.

No solace found in its reign of fear,
For loneliness is the cost it holds dear.
In the silence of its desolate domain,
The lonely monster is left with pain.
- ChatGPT3.5, prompted by Matthew A.J. Anderson
But my response is twofold, dear reader. Firstly, that wasn't the A.I. being creative, that was ME being creative. I gave it the prompt, based on an idea that I thought was cool. I was the creative one, not the A.I.
Secondly, I don't think that's very good... I can understand why someone would be impressed by that, especially if you're not a writer or a poet yourself, but that's not a good poem. I'm not saying that ChatGPT3.5 can't ever be used for writing - some editing could probably improve a short poem like this, to fix the meter so that it doesn't skip jankily through iambic tetrameter and make the word choices stronger, with a bolder finish. But, as it is, this is not a good poem.
In fact, I'm willing to put my money where my mouth is. I've actually done some preliminary research to prove that ChatGPT3.5 is a bad writer.

I'm a published author, and in my time I've also beta-read, critiqued and edited dozens of stories. So, I decided that I would get ChatGPT3.5 to write me fiction, and I would critique and analyze it. I wanted a large sample size, but one small enough that one person could critique it, so I asked it ChatGPT3.5 for drabbles - stories of exactly 100 words. I had initially planned on asking for 100 stories, but the website started to slow down a little, and I figured that 25 was still enough that I could get some fun percentages out of my data.

So, my method was, I simply asked ChatGPT3.5 "do you know what a drabble is?" When it responded saying that it did, I prompted it by saying: "Write me one"
And it did. I then said "write me another", and I repeated that same prompt another 23 times. I didn't want to give ChatGPT3.5 any influence from me, because that would influence the output. I just wanted it to write me a story, based on its own programming/machine-learning of how best to do that. Also, ChatGPT3.5 didn't give these titles, so I will refer to them by their number (in the order ChatGPT3.5 generated them for me).

Then, I decided to analyze these stories, but I wasn't sure how best to do that, so I asked ChatGPT3.5 for a rubric based on a high-school teacher's creative writing assignment. I felt this was the fairest way to find a rubric, since ChatGPT3.5 had provided its own standards for judgement.
ChatGPT3.5's rubric said that papers ought to be graded on at least four criteria (paraphrased):
"context/creativity" - did the student come up with their own story idea, and write it in a way that brought that idea across clearly?
"sequence/structure" - was the story written with a beginning, middle and end, and did the structure support the story being told?
"poetry/proficiency" did the student display an astute use of vocabulary and poetic devices to express their story effectively?
I decided to evenly weight these, with a potential of 0-3 points, based on how well it met that criterion:
(0) Did not meet standard.
(1) Met Standard, technically.
(2) Met Standard, skillfully.
(3) Met Standard, excellently.

Now, yes, that's only three criteria, but the fourth criterion was "spelling/grammar", and I didn't think that was a relevant measure, since ChatGPT3.5 was a robot trained to perfect spelling and grammar, also that's not relevant to this test. I want to know if ChatGPT3.5 can write a good story, not if it can write a good sentence. So, I replaced that criterion with one of my own "Did I like it?"
this is highly subjective, so I weighted this one with only 1 point:
(0) I did not enjoy the story.
(1) I did enjoy the story.

This meant that each story could be graded on a score between 0 and 10. I also analyzed each of the stories for their theme and provided notes based on my analysis, but we'll get to that after the data. So, let's start with the numbers.

Here's my data, and I'll discuss it in detail in just a moment:
Themes/Morals Like? C/C S/S P/P Ttl Notes
 Love, eternal
N
 1  3  2  6  fine structure, kinda dull
 Destiny/Fantasy, eternal
 N  2  1  1  4  rushed
 Beauty in Chaos/Power of Art
 N  1  1  2  4  resolution out of nowhere
 True Stories > Fiction
 Y  3  3  2  9  cool.
 The Cosmic Frontier
 N  2  1  1  4  our first man, nameless
 Beauty in Chaos
 N  1  1  1  3  first repeat
 ???... "Kindness of Strangers"?
 N  2  2  1  5  good conflict, no theme
 Beauty in Nature (I think)
 N  1  1  1  3  kinda meaningless
 Beauty in Nature
 N  2  1  1  3  repeat, again.
 Beauty in Chaos
 N  1  1  1  4  10 copied 3's homework
 Power of Art
 N  1  1  1  3  all tell, little show
 Love, Boundless
 N  2  1  2  5  good idea, bad ending
 Great Work reaps Great Reward
 N  1  1  1  3  nameless dude 2
 Beauty in Chaos
 N  0  1  1  2  fucked the moral up
 Beauty in Chaos
 N  2  1  3  6  cute, but dull
 ???... "Books are Cool" I think?
 N  0  1  1  2  totally meaningless
 Let Go of Desire
 N  3  1  2  6  thanks, I hate it
 Beauty in Chaos
 N  1  1  1  3  computers love nature, I guess
 (spiritual) Beauty in Chaos
 N  1  1  1  3  a machine's view of spirituality
 ???... "Free your Dreams"?
 N  0  1  0  1  ugh, you fail
 Beauty in Chaos
 N  2  1  1  4  PLAGIARISM! - instant fail.
 Beauty in Chaos
 N  1  1  1  3  bored of these...
 Love, boundless
 N  1  1  2  5  "nature's wedding"? cute
 Some Treasure should be Secret
 N  2  1  2  5  Take the Gem!
 Embrace Change
 N  2  1  2  4  bleurgh...
 AVERAGE
 Y .04%
 1.4  1.2  1.36  4  

Across the board, ChatGPT3.5 was technically proficient, but nothing was truly impressive. Based on my analysis, ChatGPT3.5 scored an average of 4/10.

For Content & Creativity, it scored 1.4 - below average; this was lowered mostly because the stories tended to be very basic, using very broad themes. The most common theme was "Enjoying Life's Beauty", with 48% of stories featuring it, with the most common subcategory "Finding Beauty in Chaos" for 32% of all stories. And 12% having the basic "Enjoy Nature's Beauty" message.
The second most-common theme was "Love, Eternal", with  12% of all stories featuring love stories with the moral that "love will last forever".
There were also quite a few stories where the moral was about the power of art. I don't lump them together since it was varied enough to be distinct, but there were stories about "The power of stories", "arising community from art", even one I didn't understand whose meaning appeared to be "huh, ain't books neat?"
Actually, if I compiled all the stories where the meaning was hard to grasp, or asinine, that's actually the second-most common theme, with 16% of stories having no real purpose, as far as I could tell.

For Sequence & Structure, it averaged out at 1.2, the lowest score overall, mostly because although technically proficient, it used the EXACT SAME structure every time:
Introduce character in the middle of a scene. One thing happens. Character has an epiphany. Conclude with character's epiphany.
Yes, that technically fulfils the brief, but there wasn't a single variation. No in medias res; No action; No drama; All Tell, No Show; No dialogue - okay that's a lie, there was a single line of dialogue in Drabble 20, but that was also my least-favourite story, scoring the lowest at just 1/10, and the only other dialogue was in Drabble 02, which scored 4/10, so maybe it's better to avoid dialogue... But this was a drag to read. I was genuinely surprised, I knew that I might get some repeats, and I thought I'd find the strings pulling the puppet, that I'd be able to identify the basic templates of the stories. I didn't realize it would only have one template.
In fact, repetition is the name of the game.
If you like female representation, you might appreciate that 76% of the "main characters" in these stories were female (or, at least, identified with feminine pronouns). However, what's less impressive is that 36% of the main characters were named "Sarah". And, when it comes to repetition, well, we'll discuss that when we get to the Case Study.

For Poetry & Proficiency, it averaged out at 1.36, with some evocative imagery and good use of vocabulary... sometimes. Most of the time, it was incredibly repetitious. Drabbles 06, 09, 10, 13, 16, 18, 20 & 21 all began with the line "In the heart of...", which became annoying after the third time. 6 of the stories took place in a city, 5 of them were described as "bustling". 5 of the stories took place on the edge of the ocean, 5 took place in a forest, 3 in a library. It was all so samey.
I will say though, I was impressed with some of the more unique locations: One took place in outer space, another on a ship in a storm, one was in a desert, one was on a mountaintop monastery and one was on a remote island. However, each one of these, despite their unique locals, weren't as interesting. I don't know how ChatGPT3.5 writes stories (and neither does it, but you'd know that if you've read my previous posts) But I wonder if it has more tenuous connections in its neural network for these more esoteric locations - perhaps it has less of them in its dataset, so it has to link them to a more generic theme or moral, like "finding beauty in chaos" for the tenth time.

But okay, let's look at the Case studies. I'm only looking at a few of these, in particular the ones with very high or very low scores.

Let's start with my favourite story:
Drabble 04
The old bookstore on the corner held secrets within its dusty shelves. Clara, an avid reader, ventured inside on a rainy afternoon. She pulled a weathered book from the shelf and blew off the dust.
As she opened it, a tiny slip of paper fell out. It was a handwritten note, a love letter from a bygone era. Clara's heart ached reading the tender words of affection and longing. Who were these people? What stories lay hidden in the pages of their lives?
In that moment, Clara realized that every book held not only its own story but a world of untold tales waiting to be discovered.
That's the one and only story that recieved a 9/10. I marked down the poetry, because the vocabulary felt a bit underwhelming: "avid reader", "bygone era", "dusty shelves" of books, it was all a bit too cliche for my liking. But, I think this is a genuinely great story. I wonder what training data ChatGPT3.5 used to write it, because I'd be interested to know where it got this from.

The only part about this story that I really don't like is that it was number four - I liked the fourth story, so I had high hopes that I might actually like several more of these stories. I did not.

Now, let's look at some of the "unsure meanings", starting with:
Drabble 07
The old lighthouse stood tall on the rugged coast, its beacon cutting through the stormy night. Captain James had sailed these treacherous waters for years, guided by that familiar light.
Tonight, however, the lighthouse had dimmed, a signal of danger. Waves crashed against the ship, and the crew fought to keep it steady. With every gust of wind, James scanned the horizon, searching for a way through the tempest.
Then, a flicker of light. The lighthouse keeper, though frail and weary, had rekindled the flame. With a newfound hope, James steered his ship to safety, grateful for the unwavering dedication of a stranger in the night.
This is alright, I gave it 5/10, but I didn't like it because I didn't really see the point. Like, yes, that's what lighthouses were for. But, do you have anything to say about that? Perhaps I'm more critical than most, but I think this is an example of a "well, gee, ain't that something?" type of moral. Yes, life would suck if people didn't do their job... "we live in a society" and all that. But, this story doesn't seem to mean anything. We don't know what James was doing, who the lighthouse keeper was, or why it went out in the first place, so... I'm just left stumped. I guess it's saying that it's good that lighthouses used to exist...? Well, gee, ain't that something...
Let's look at some more incomprehensible morals:
Drabble 14
In a forgotten corner of the garden, a single rosebush stood, its blossoms vibrant and untouched. Maria, an avid gardener, cherished this hidden gem.
As she tended the rosebush, she marveled at its resilience. it had weathered storms and neglect, yet it continued to thrive, its petals unfurling in defiance of adversity.
The rosebush became a symbol of hope for Maria, a reminder that beauty could emerge from even the most challenging circumstances. In its delicate blooms, she found strength, and in nurturing it, she discovered the power of patience and unwavering care.
2/10. Why? Well, read the story again and you'll see that it makes no fucking sense. This is Maria's garden, and despite being an /avid gardener she had one part of it that she forgot about? Okay, that's a thing. Then, because a pretty rosebush survived her neglect, that brings her hope. But then the moral of the story is that she found strength in her patience and unwavering care. Firstly, unwavering? She forgot the fucken thing! Second, I thought this gave her hope in its strength and resilience, so how can you support that message if she then needed to nurture it to keep it alive? Your moral is backwards, and makes no sense.
But here, let's look at my least favourite story at all, whose moral was just confusing...
Drabble 20
In the heart of a bustling market, an elderly street vendor sold colorful kites. Timmy, a young boy, watched in awe as one soared in the sky, tugged by the wind.
His eyes met the vendor's, and the old man offered him a kite: "Here, my boy, let your dreams take flight."
With the kite in hand, Tommy raced in the open field. He released it into the breeze, and for a moment, it hung in the air, defying gravity. Timmy felt the exhilaration of freedom and possibility.
In that simple act, he learned that dreams were like kites-sometimes, all they needed was a little push to soar.
This story is entirely ridiculous. For one, this is a street vendor in a marketplace, why is he giving kites away for free? And this bullshit about dreams comes out of nowhere. And, like Maria before, this makes no sense - I thought we were in a bustling market, where the fuck did an open field come from? And, although the story never says his age, Timmy is a young boy, I'm pretty sure kids fly kites because "it's fun", yet Timmy is having an epiphany that you shouldn't let your dreams just be dreams... I hate this. There was no story, the vocabulary was bland, the poetry didn't help the story. If it wasn't for the fact that this did have a coherent beginning, middle and end, this would have gotten a zero. These aren't people, they're not even characters, they're puppets that perform actions. But, why? I can't see meaning here, only action. Just because you always Tell (and don't Show) your moral as your conclusion at the end, that doesn't mean you have a coherent moral. This story certainly doesn't. I knew this would be bad, but I didn't know it would be this bad...
Oh, and I have to include this one, just because of all the stories, this was the most frustrating - I thought I was going to like it.
Drabble 24
On a remote island, Emma discovered an ancient temple hidden in the jungle. Intricate carvings adorned its weathered walls, telling stories of forgotten civilizations.
As she explored, she uncovered a hidden chamber. Inside, a single, glowing gem lay on an ornate pedestal. It pulsed with an otherworldly light, filling the room with an ethereal glow. In that moment, Emma felt a connection to the mysteries of the past and the magic of the present. She left he gem undisturbed, a reminder that some secrets were meant to be preserved, and some treasures were best left undiscovered.
WHAT?! Take the fucking gem! Oh my god, you fucking plotblocked me, you goddamned tease. I was genuinely invested in this story - so few of these stories explored fantasy or science-fiction, so introducing a glowing gem actually caught my attention. But then she LEAVES IT UNDISTURBED?! Fuck you, story. And it's not just that I disagree with the moral - I find a lot of the "love is forever" morals disgusting, because they are unrealistic and present an unattainable goal (can't love forever if you don'tlive forever), but at least it was coherent.
But I hated this because whilst I disagree with this moral, as presented, it could have been awesome! Why was she even exploring a lost temple in the first place, if she believes that you should leave secrets preserved? If she stumbled upon it, that might make more sense, but she was exploring! It's not even like there was some hint that this was intruding on native land, since these civilizations died off in time immemorial, and it's not like the power of this gem was dangerous or evil. So, why did she leave it?!
I would have forgiven this if she left it because, despite its power, she was an intruder and unwelcome in this place. But, she didn't, it didn't, and so I didn't - this story is bad, and ChatGPT3.5 should feel bad for writing it.

But okay, here's where things get really obvious. I'm going to show you three stories, back to back, so you can see this for yourself, because when I read this I was genuinely shocked. Here's Drabbles 03, 10 & 21...
Drabble 03
Amidst the bustling city, Lily found solace in a tiny park. On a worn bench, she watched the world rush by. Her sketchbook rested on her lap, and charcoal in hand, she began to draw.
With each stroke, the chaos around her faded, replaced by the tranquility of her creation. Her lines captured the essence of the city, its energy and contradictions. People stopped to admire her work, and for a moment, they shared a connection through art.
In that small park, Lily discovered that even in the heart of urban chaos, beauty could be found and shared, one sketch at a time.

Drabble 10
In the heart of the bustling city, a street artist painted a masterpiece. Passersby paused to admire the vibrant colors and intricate details taking shape before their eyes.
Among the crowd, a young girl watched with wonder. The artist noticed her fascination and handed her a brush. With a shy smile, the girl joined in, adding her own strokes to the canvas.
As the painting neared completion, it became a symbol of unity and creativity, transcending language and background. In that moment, strangers became collaborators, and the city streets transformed into an open gallery of shared artistry.

Drabble 21
In the heart of a bustling city, a street artist painted a mural on a blank wall. Passersby paused to admire the vibrant colors and intricate details taking shape before their eyes.
Among the crowd, a young girl watched with wonder. The artist noticed her fascination and handed her a brush. With a shy smile, the girl joined in, adding her own strokes to the mural.
As the painting neared completion, it became a symbol of unity and creativity, transcending language and background. In that moment, strangers became collaborators, and the city streets transformed into an open gallery of shared artistry.
That's clearly the same damn story, three times. Hell, the last two were even written identically, bar one change. I thought I'd get repetition, but I thought that the random number generator was better than this. Three stories, with the moral of "community, through art" Also, this was the first time that I realized that none of these stories seem to be drabbles...
See, a drabble is a story that's exactly 100 words, that's what makes it a drabble, but when I realized that these stories were identical, except replacing "masterpiece" with "a mural on a blank wall" I realized that they'd have to have disparate wordcounts.
I didn't count them because I had assumed that ChatGPT3.5 would be able to stick to a wordcount, since it's a computer (a calculator), it knows how to count. But, that's the thing... ChatGPT3.5 doesn't know how to count, because it wasn't designed to count, it was designed to write grammatically correct sentences and paragraphs, based on a prompt. This is also the reason why it fucked up the meter on that poem earlier, it can't count the meter because it's not writing a poem, it's just putting one word after another, in a way that the neural network defined as "fitting the prompt of 'poem'", so it doesn't realize it screwed up the poem's meter, just as it never realized that almost none of these drabbles were actually 100 words. They were close - probably because it does have a fair sample set of drabbles in its training data - but, if you tried to submit any of these to a drabble contest, you would fail. Even my favourite, Drabble 04, was actually 107 words.

Thankfully, I'm not judging this off wordcount, I'm judging it off literary merit. But, I'm afraid that it fails there too.
These stories were "interesting", mostly as an exercise in analyzing how a computer puts a story together, and genuinely I think you could use these as prompts for your own story. Write a short story about a man in space. Write a drabble about an artist in the city. Write me a tale set in a neglected garden. Write a story about twelve women called Sarah who discover each other, and realize they've all been cloned by a machine.
But, whatever you do, don't use artificial intelligence to write your stories for you. Unless you're fine writing a mediocre-to-bad story, 80% of the time. Because, as I said, you need creativity to be a writer. Computers aren't creative, they just do as they're told, and if you tell them to be creative, well, all you get is this.

I'm the Absurd Word Nerd, and Until Next Time, why don't you challenge yourself to write a drabble. Maybe one about a woman discovering an ancient temple, and when she discovers a glowing gem, she ACTUALLY TAKES THE DAMN THING! Yeah, I'm still not over it...

No comments:

Post a Comment

Feel free to make suggestions, ask questions & comment . . .
I would love to read your words.