A long time ago, in a galaxy far, far away... Audiences were stunned by Darth Vader's words to Luke: "No. I am your father." It's a classic plot twist. Done right, it has a lot of impact. Done wrong, it's predictable and boring.
Lately I've wondered about such twists and turns in the stories we digest. There's not much that surprises me anymore. My husband and I watched Thor: The Dark World last night, and... I was disappointed. Don't get upset when I say that Loki is predictable, because there are some fantastic moments in that film and almost all of them come from Tom Hiddleston. (In fact, he's the only character to show some real emotional depth.) But it's true. Loki is really predictable.
One could argue that Star Wars was predictable, but since I first watched it as a little kid, I was thrilled by the worlds and characters, and what I see all the time in other things now didn't feel so cliche then. The mysterious and powerful mentor must die for the hero to grow into his own. The rogue runs to a friend for help and is betrayed. The bad guy turns out to be the hero's relative. Put them together and you either have an amazing story or a really lousy one.
How can this be?
It's all about storytelling. It is our jobs as writers to tell the story properly. The plot thickens. It twists. Everyone knows and expects this. What matters is how this is shown. You can do this three ways:
1. The obvious way. It's the lazy way; this is how, say... Catching Fire comes across. Hints are dropped every five seconds just in case the reader misses the first hundred clues. The protagonist is miraculously stupid even when literally everything points at the truth. (Really, Katniss? How can you be so blind?!)
2. Subtle foreshadowing. This is tricky, because if you don't heed the word 'subtle' here, it will turn into the obvious. A good example comes from what now seems like an old film, The Sixth Sense. It was talked about to death after everyone had seen it, because while the clues were there, they were easy to overlook. (Since it came out over ten years ago, I feel no shame in sharing spoilers.) Dr. Crowe never moves anything, and doesn't ever have a direct conversation with anyone but the kid. (The Anniversary scene was brilliantly acted, by the way.) Then when you find out he's dead, you suddenly want to go back and watch the movie again. And you realize, all the signs were there.
3. Present it right away, then let them watch it unfold. My favorite book--or rather trilogy--right now is Mistborn. There is a huge 'secret' throughout that's easy to deduce once the third book starts, but as soon as you notice, instead of being put off or disappointed by it, you want to see just where the author is going with it. And as all the little pieces tie together, you marvel at how amazing the whole story was, and how even the expected had an unexpected twist. (Then you cry so hard that you can't even see what you're reading anymore... ahem.)
Now you may say, "But what about random twists? Out of the blue, for no reason?" Random stuff happens in real life, but not in fiction. It needs to serve a purpose. Even a book that claims something random just occurred proves a point while doing so. Pay attention to what you're doing, and don't ever force the story to change just because you think you need a plot twist. That's the most important thing. Don't force it. Your audience can tell.
I wish you luck. Now go write.