How to Succeed with Failure (in Writing)

Most stories are built on conflict. It doesn't have to be violence - that is to say physical conflict - it can also be logical conflict; psychological conflict, or social conflict. Maybe it's the cute girl the guy wants to impress; maybe it's the evil monster the hero wants to kill; maybe it's a killer the detective wants to find. But, in simple terms, stories often start with a problem, and end with a solution. Not always, but often. It's one of the things I truly love about Horror - it's one of the few genres where the main characters can fail miserably, and the audience can still leave feeling satisfied.
However, stories often aren't just about one problem. The guy wants to impress the cute girl, but she's rich and he's poor; he's been cursed to look like a beast & she has a husband. The hero wants to kill the evil monster, but he isn't strong enough; the monster lives far away & the monster commands an evil army. The detective wants to find the killer, but the murder weapon is missing; the room was locked when the crime happened and everyone else just thinks it was a suicide.
These myriad problems are put in the way, and the main character has to solve them, one by one, before moving forward.

So far, so standard, if you're a writer yourself none of this is new. But, here's the rub. The main character has to solve these problems to continue the story. If they don't, the plot can't move forward. And if you're not careful, this can lead to a sense of inevitability.
Problem? Solution; Problem? Solution; Problem? Solution...
It can feel like going through the motions.After all, if your main character fails, then the story stops progressing. I mean, if you're threatening to kill the main character, but I'm only halfway through a book, I know my main character can't die since I've got a whole lot more pages left.
The best example of this is in action movies, where the hero is slaughtering his way through a bunch of bad guys. If he loses, then he dies, the story ends. So, he can't lose. But, it doesn't even need to be death. In a murder mystery, one MUST find the missing clues, and one MUST interpret them correctly, otherwise they can't solve the case - there's no other choice, because if you fail to find a necessary clue, then necessarily you can't solve the case. Or, if something comes between two lovers that drives them apart, we know they MUST get over it before the story ends, because romance stories thrive on happy endings - failing to overcome that conflict makes the whole story fail.
So, your character can't really fail... but, when your character comes across problem after problem, and manages to solve it no matter how difficult, this becomes a problem known as Plot Armour.
No matter how devastating the problem, no matter how many times the Earth needs to be saved from Doom, the Hero wins, because they have to. As soon as the audience realizes this, the story loses all tension. We know he can't lose, so watching them win isn't entertaining, it becomes a foregone conclusion. It's one of the most frustrating parts of formulaic stories - having the same story structure every time means the conflict stops feeling like conflict.

This idea is best illustrated by One-Punch Man, a webcomic, turned manga, turned anime. I highly recommend it, as it's quite funny, but the point of the story is simple. Saitama is a superhero, called One-Punch Man, and he is the strongest superhero in the world, so powerful that he can win any fight, with just one punch (hence the name). For this reason, he's also one of the most boring superheroes in the world, because he can't lose any fight.
Thankfully, neither the webcomic nor media based on it are action stories, they're comedies and character pieces, exploring the character and those around him as he struggles with a sense of ennui, feeling empty despite being literally the best in the world and how other characters react to an unfathomably unstoppable hero. See, by being unable to fail, the physical conflict stops being a conflict - so, the story is instead about social and psychological conflict.

Now, your hero probably isn't able to win fights with a single punch, but if you're not careful, it can feel like it. If you're a newer writer especially, you may think that if your hero keeps winning, but you don't want them to lose, then the solution must be to make the challenges more difficult. Escalate: If last time, the bad guy threatened his family, next time his whole town should be in danger. If last time, she killed a tribe of monsters, next time she should fight a whole army. Did your hero save the world? Well, next time, save the galaxy.
The problem is... this actually makes the conflict less impactful, not more, because no matter how much they struggle, they HAVE to win, since you've stacked the deck in such a way that failure is impossible to read (or, in some cases, write). It's what I call the Shan Paradox - by continually escalating the stakes, you inversely lessen the tension, because the more the stakes, the more impossible it becomes to lose, since that kind of devastation is impossible to overcome.

So, what's the solution? Well, the answer to that, I think, is obvious... you need failure.

I think the best way to explain this is with Pen & Paper RPGs (Role-Playing Games). You've probably heard of Dungeons & Dragons, but there are many more and better RPGs, but whether you have or not, the important thing you need to know is that when playing a Pen & Paper RPG every player gets to be a main character (except for the Game Master, who gets to be everyone else). In an RPG, your character can die, which holds a lot of weight because both the GM and the Player wants the game to keep going. The GM either spent a lot of time creating the campaign story (or money and time setting up a purchased module) and the Player usually spends a fair amount of time creating their character and writing up their character sheet, so  both of them are invested and neither of them explicitly want the characters to lose. But, believe it or not, loss and death is an important part of the game. See, if the player knows they can't fail, they lose interest in the challenge - this is a game after all, and games are challenges defined by having both a Win state, and a Lose state.
Because none of the players want the characters to die, most Pen & Paper RPGs are played with dice (or coins, or cards, or timers), so there's a random element that can strike when you least expect it, and prevents either the GM or the players from making the game too easy. Failure is a looming threat. Oddly enough, whilst nobody wants to fail, failure is the critical element of the game that drives it forward. The fact that you might Lose is what makes it so much more satisfying when you Win.
But, not every dice roll is the final roll of life and death... in fact, games would be frustrating if you randomly and suddenly lost the game every few rolls. You roll the dice for any challenge in the game, whether that be combat, investigation, travel, coersion, romance, stealth, athletics or magic - and remember that's challenge, not action, it would be ridiculous if you had to roll a dice for every step to make sure you don't trip... but my point is, it means that in order for your character to die, and completely lose the game, you'd have to either make a bad decision, or engage in an encounter where you might not succeed; then you have to fail several times during that challenge's process, enough to weaken your character's chances; then finally, most games even have death-saving throws, meaning you'd need one final bad luck roll to actually kill your character. And hell, even if your character does die, some games (and some GMs) offer the player the option to create a new character, or use a backup character to continue the campaign. This is still a hell of a loss, because they lose that character and that character's experience (sometimes in more ways than one, if the game's mechanics use experience points). Some games are harder than others, but very rarely do they randomly kill players for no reason.
But, what happens if your character fails a smaller moment? Say, lockpicking, that's a useful skill. Say your character wants to get into a locked house, and there's a plot-critical element inside the building, but they fail their lockpicking check - roll a 1 or something. What happens now? Does the story end?
Of course not. The player now needs to find another way in. They could try another door, but that's just the same challenge again. They could instead try investigating the surrounding area, to see if there's a hidden key somewhere, or they might have to find the owner of the property and see if they can pickpocket the key from them. They could perhaps attempt to climb the building to see if they can get in through the chimney, or a large window, or a vent, but if none of that's available, they might simply have to just break and enter, bash the door down or break a window, even though that may alert people nearby, and will definitely come to the owner's attention when they get back. There's more than one way to skin a cat, as they say - but more importantly than having multiple solutions, there's also multiple failures for every problem. If the character attempts to pick the lock, and fails, they may simply need to try something else, or they could break their lockpick tools, meaning they'll need to find more. If they break a window and jump through, they may hurt themselves, but more importantly, they may have the police after them, or worse - if this is a secret/criminal building they're breaking into, they may have a hitman or criminal gang chasing after them. All of these add potential conflict to the role-playing game, which makes the adventure more exciting
This is just one example, but the point is, the failure doesn't have to stop the story dead, you don't have to accept a failure, and more importantly failure can actually make a story more interesting. If you talk to an experienced RPG player or GM, they will often have great "war stories" of some of their most fun gaming moments, and I guarantee that their best stories won't be about all the times they got perfect dice rolls... no, they're often tje stories about how they had to come up with something new, clever or silly after they failed, or about how a situation got completely out of control because they didn't solve a problem correctly.

This all is great information for writing linear stories. Even though a linear story isn't a game, that potential for failure is a necessary tension for most stories. Obviously, not everything is a challenge, and you probably shouldn't have your character always fail (unless you're writing a slapstick comedy), but you can have them work their way towards success and fail along the way sometimes. Hell, that would be a fun writing challenge... write a story as though it were a one-player RPG, and rolling a die or flipping a coin every time they attempt a difficult challenge, to see if (and how well) they succeed or fail. That's not necessary for writing a story, but my point is you can, and should, let your hero fail along the path of their adventure.
You might think it makes a hero more of a badass to get out of a fight without a scratch; solve a mystery without getting anything wrong or fix a problem without any negative consequences (and depending on what you're writing that can work), but readers tend to be more invested in a hero they can empathize with. Since we often struggle in our own lives, if your hero struggles it means the reader can feel their pain along with them, and feel their success along with them as well.
Consider, for a moment, Action. You might think losing a fight means death, but not neccesarily. Not every fight is a fight to the death, what if you don't lose your life, but lose an arm instead? Or a foot? Or, what if they get cut across the throat, and from then on, have a scratchy, hoarse or otherwise damaged voice due to an injured larynx, or vocal chords? Disability isn't the cultural stigma it once was, and even if it is in the period, setting or world of your story, if anything it can make your story more interesting. Prosthetics are amazing, eye-patches look cool and scars often mean that your character is wearing a part of their backstory on their skin, it's not something you have to avoid.
This is actually a big problem I had when I was a younger writer. I didn't see the point of combat at all, because I knew the character had to win, and I didn't want them to get hurt - and I didn't realize until later that it was that last part, a desire for them not to get hurt, that was actually holding me back. Nobody plans to fail, we might have contingency plans, but rarely is our initial plan "I'll do this, it will fail, then I'll try something else", and if you're not used to writing, then adding failure into your plot can feel forced. But you don't even necessarily have to force it, it can happen naturally...

I tend to write scenes by envisioning them in my mind, and describing it as thoroughly as possible to place readers in the scene, and I remember writing one story wherein a character had to attack an enemy in close-quarters, in their bedroom, with a cricket bat. As they raised the bat above their head, I realized that (in my room at least) I had a pendulous ceiling fan with a light on it, and if I swung a bat over my head it would hit the light, so I added "the bat narrowly missed the ceiling light as he swung". As I read what I'd just written, I thought to myself "Gee, it's lucky he missed that light, if he had it would have sucked, he'd have been covered with shards of glass, that sounds horrifying..."
Then I had an epiphany, dude, you're a horror writer, why would you want to avoid something horrifying? So, I rewrote the scene such that he smashed the ceiling light, the shattered glass falls on top of him and the whole scene became more impactful as the monster approached him, crunching shards of glass under its bleeding feet - it was awesome.
It changed the story, but for the better, success is boring, success is what we expect, so when something fails it can snap us to attention since now we - along with the character - are left wondering how the story can possibly go forward from here, or what more could possibly go wrong.

And it doesn't have to be physical damage, you don't have to marr or maim your character, you could just hurt their pride instead. A lot of comedies thrive on characters failing, so you can use failure for a comedic moment, but if you want something more dramatic you can go that route as well.
Consider, for a moment, Mystery. Yes, your character needs to interpret clues, but what if they follow a lead that's wrong, but only slightly wrong. Even people who aren't fond of mysteries are aware of the red herring, a clue that leads you down a bad path, this is actually a great example of failing in a story. You discover that there's blood at the crime scene, and one suspect has a cut on their hand! But... it turns out they filled in for the cook a day before the murder, and they're bad with knives, which is why they cut themselves and bled at the scene - dead end, it was just a red herring (a misleading non-clue) no help whatsoever.
But, a clue can still help the mystery, even if the detective gets it wrong. A detective might find bindings on the victim, and assume they were tied up by their killer, so they follow the clues, find out where it was purchased, and realize the ropes belonged to the victim's wife... who used them in their nightly roleplay. That clue might seem like a dead end, but what if the killer is some religious or righteous prude, who killed the victim because they saw their nightly activities as deviant? In this way, although their assumption lead to failure, their failure helped lead them to success... Not so much a red herring, as a fox-tail... you didn't catch the whole fox, but you got enough to lead you somewhere (I thought "fox-tail" worked, since apparently "red herring" was initially a fox-hunting term... don't blame me, I didn't come up with it).

Or, what if the failure is finding the clue in the first place. What if there are clues at the scene, but your detective can't find them, so decides to interrogate all the suspects instead. This can make the story more compelling as you get to hear the characters' alibis and personalities. And heck, if they don't find the clue, maybe the suspect who left it - the killer - might be brave enough to see if they can steal it before it gets discovered. This could lead to them acting suspicious in a different way, and/or leaving more and different clues in their wake. Your detective doesn't have to be a Sherlock Holmes type, who can solve any mystery (unless a pretty lady is involved). Your guy can make mistakes, especially if you're not writing detective fiction, but are instead having a smaller investigation or treasure-hunting plot in the midst of your broader adventure.

And hell, I can go on for hours talking about how you can use failure to add plot complications which improve your story, because honestly, failure can be a broad spectrum... hell, success can be a broad spectrum. Ever heard of a Pyrrhic Victory? You couldn win the battle, but ultimately lose the war.

The point is this... it's easy to fall into the trap of thinking that stories are about watching someone win, again and again and again. But, that's not the case, at least it shouldn't be, and if you write a story like that without the necessary finesse it can alienate your audience. As I have said already, to win at everything and never fail is inhuman.
To err is human; we all lose sometimes... not only will we all eventually die, but we also sometimes get badly hurt, get it wrong or cause problems because of our mistakes. But, our story doesn't end every time we fail, we keep going, but we might have to change... to develop, or adapt, to a new paradigm caused by our failings. That's how we develop as people, and it's how stories develop into legends.

I'm the Absurd Word Nerd, and until next time, I hope that the next time that you fail, you have the strength to get back up and try again, or find a way to move past it.