The obvious reason is that RPGs are based on the works of Tolkien and of adventure-fiction authors.
(Of course, those bodies of fiction may, in turn, be quest-driven because a quest is the best kind of story for a fantasy world, since a quest has the ability takes the reader through lots of locations – and it’s a definitive feature of fantasy worlds that they have lots of locations that are worth reading about – but that’s not the point of this article.)
But I think the reason that the quest format is so perfect for RPG storylines has to do with holding the players’ interest. When we are reading a story about characters who are on a quest, an interesting thing happens. We know where they’re going, but we don’t know what they’ll meet along the way, what the complications will be, or if they’ll succeed. So, we start out with enough information to keep us interested (they’re on their way to steal their treasure back from a dragon!), but not enough to dispel the suspense or excitement from the story.
This phenomenon, by the way, should also characterize a healthy campaign: the players know (basically) where their characters are going – whether they’ve been told all about it by an NPC or whether the GM was feeling crazy and let them create the “where they’re going” part themselves – but they don’t know the details and surprises that will happen on the way. Those are up to chance and the GM.
Quests tease us by not quite letting on what the story is going to be.
But another reason quests make for great gaming is that they allow the players to know who the characters are. When we are reading a story about characters who are on a quest, we know what they want, where they came from, and what they’re like – all in addition to knowing where they’re going. This makes them interesting to us, so we want to stick around for the uncertainties they will face. Similarly, players want to play characters they’re interested in. And a quest format allows players to know not just who their characters are, but probably where they came from, and (if you’ve given them some creative freedom in this) where they’re going. Knowing where a character came from is good, but knowing where a character is going or trying to go is real engagement.
Quests set you up to know a lot about your character so you’ll care about how he struggle through all of the uncertainties and mysteries that are to come.