The Legend of Zelda: Ocarina of Time
Ok, let's get this one out of the way. I almost didn't include The Legend of Zelda: Ocarina of Time, purely out of principle. What can be said about this game that hasn't already been said? Of course Ocarina of Time has good music. What Zelda game doesn't? But is the game's music vital to your enjoyment of it? If the game's score were taken away, would the game suffer?
As much as I hate to say it, I have to answer "yes." Not only does the game use unique songs to give each major character and location in Hyrule its own unique personality, the ocarina itself makes knowing these tunes vital to progressing the game's narrative. Music isn't just a set piece in Ocarina of Time; it's a tool.