The ambiguous ending can be the hardest to write because it can easily fall flat. It can be a thin line between leaving the ending ambiguous and leaving it confusing. By this I mean that a good ambiguous ending can promote interesting debate with people pitching in different ideas, it goes wrong when it leaves readers confused.
An example of a good ambiguous ending would be the Usual Suspects where we’re are lead in the direction as to who the mysterious arch-villain is, however, it is never explicitly stated who it is or if the arch-villain is actually real or part of the misdirection. We can say for certain that Kevin Spacey’s character is not a good man and that he’s an expert liar and, oh… there’s the problem. We’re narrated through a story by a character we come to realise is an out and out liar so we assume that he is the arch-villain but we couldn’t say for certain that he is that man because we can’t say that man exists rather than being part of his fabrications. This is what makes the ending ambiguous although it has entered popular culture as definitive. All we can say definitively is that the character is a liar being interrogated in relation to a crime and a lot of his facts match up with items around the office he’s being interrogated in. Despite this lack of certainty it’s a good ending, in fact the lack of certainty plays into it being a good ending because the audience has the double realisation that Spacey’s character is lying and might be the mysterious man but they can still speculate on how much he made up.
If the story had been a straight-forward one about a crime boss that bumps off a bunch of criminals then the ending would’ve been a lot less memorable and a lot less debated. There are people who will watch The Usual Suspects and think they definitively know the answer, there will be others who debate the answer and might even analyse the film and there are others who will be completely confused. But the pieces were all there for them to assemble as they wished.
Often the ambiguous ending falls down when it’s too ambiguous. By this I mean the pieces aren’t there for the reader to assemble. An ambiguous ending that is too ambiguous often has a tendency not to feel like an ending and frustrates the reader rather than making them enthused to read more. At these moments it helps if as the writer you have a definitive answer, or at least an answer you want to nudge people towards.
At this point you might observe that I keep mentioning Beckett who was very ambiguous. Firstly, I might observe that at the end of a Beckett play it feels like an ending, even if it feels like an ending where the characters fade out rather than a sharp ending, but an ending nonetheless. Secondly, I might observe that Beckett’s plays are short so, simply put, the time invested in less. A short play with a very ambiguous ending is a different proposition from a five hundred page novel that never seems to have concluded anything.
Finally I might observe that Beckett wrote plays and when a play is staged there invariably ends up being some form interpretation put on it by the director or cast or critics. Plays are usually written to be performed not read so they aren’t designed to go unfiltered from writer to audience. Therefore what is purely ambiguous on the page is rarely so ambiguous on the stage because it’s difficult not to put an interpretation on something. Even if something is played ambiguously human beings automatically put motivations to other human beings’ actions which is more easily done for actors than dialogue on a page.
Due to the performance step between writer and reader a play is a slightly different creation to a novel but I would still say that the ending needs to feel like an ending not simply that the writing has stopped.
All these factors are what make the ambiguous ending so difficult, while I don’t want to be discouraging I would suggest first trying definitive endings before writing ambiguous ones. As with anything it is best to begin with foundations before building walls. Ambiguous glass walls.