Some of the most fun my players have had were improvised adventures that I just came up with off the top of my head. I didn't really know what was going to happen until it happened. The enemies were more interesting, even though I didn't have the rules and stats to utilize. It's something of an artform to do a spontaneous adventure, but one that it is necessary for any DM to be able to perform, at last to some degree. If your players are anything like the ones I've had to deal with in my years of DMing, then you know how easily an adventure can get derailed. You've worked hard on planning every dungeon, every room, every creature, every encounter. And then the players decide to turn left instead of right, and avoid the whole mess. Some would say that a good DM would find a way to get them back on track. I disagree. After all, part of the fun of playing D&D is the freedom it allows you to do whatever you want. Sure, there should be consequences, but railroading your players into a pre-planned adventure isn't good DMing. I think a good DM should be able roll with the punches, and pull a decent adventure out of his ass when the situation calls for it.
So, how does one actually do that? Well, it does involve knowing your Player Characters (PCs) pretty well. You should know how powerful they are, and what they can actually do. Even if you don't, maybe because it's a new campaign, or some of the players had to roll new characters, or there are new or fewer players now, keep in mind that you are the DM, and you can make adjustments to monsters and other elements on the fly. That is a very important concept to keep in mind. Your imagination shapes reality in a D&D session. Did you accidentally put the PCs up against a challenge they cannot overcome? Gimp it. Maybe the monster has a lame arm, and is denied some of its attacks. Maybe it has just finished off another band of adventurers, and is low on health. Maybe that trap or that lock you made too strong is really, really old, and is overcome much easier than it should be. The reverse of this is also true. If you make something too weak, you can find ways of making it stronger. The universal explanation for everything that doesn't make sense, Magic, works really well here. However, I find that it is often better to let them have the victory over a challenge you made too weak to wet their appetite for the next encounter.
NPCs are one of the easier aspects of making an adventure on the fly. If you pay attention to all the rules governing NPCs, you notice they suggest that every NPC your PCs come across should have their own character sheet, filled with stats and NPC Classes and such. I find this is largely unnecessary. If it really comes down to it, you can make a pretty good estimate of what their roll modifier will be, based on who and what they are. A blacksmith, for example, is going to be strong and stout. He will likely have STR and CON bonuses of at least +2, maybe even +3. As "normal people" tend to go into careers that complement their existing talents, it is not likely he will have a DEX bonus of higher than +1, though likely it will be average, or +0. Any bonuses to their remaining three ability scores should reflect their personality. A gruff, silent-type balcksmith would have a CHA bonus of +0, or maybe even less. Where as a friendly blacksmith who can't wait to sell you his wares might have a CHA bonus equal to his STR bonus. Is this blacksmith Rogue bait, or is he going to put the Rogue in his place? That would affect his WIS bonus. You should also know about what level he is. Basically, have an idea of how high you want his average result on a given check to be, then subtract 10 (the average roll on a d20) and his relevant Ability modifier. The remaining number is how many ranks he needs in that skill, or how much of a class bonus he needs in that save/attack. That will give you an idea of what level he would need to be in order to achieve the result you want, which will in turn give you an idea what the ranks/class bonuses for other rolls would be. This process also works for Character Classes for NPCs and creatures that the PCs might have to fight.
Monster follow a similar process, but are a little different. Most monsters you can still pull directly from the Monster Manuel/Bestiary. However, sometimes you need that custom creature to add the right flavor to whatever hairbrained idea you've come up with in the five seconds since your PCs took the wrong fork in the road. Again, this is where it is really helpful to know what your PCs are capable of, but, with the advice I gave above, you can fix any miscalculations. Have in mind the highest AC you want the monster to be able to hit, set your average to a bit below that, then subtract 10. That is it's total attack bonus for its primary attack. Repeat these steps for saves and skill checks as they come up. Don't worry too much about hard rules for any special abilities, just have them happen. If your PCs want to make a save against whatever effect it was, make up a number and have them roll against it. Keep that number in mind, though. All subsequent saves your PCs make against this creature shouldn't stray too far, if at all, from that number. First and foremost, however, let your imagination be your guide.
The adventure itself is basically window dressing. It's all appearance and atmosphere. Tropes are your friend in the spontaneous adventure. The hook doesn't have to be terribly creative or unique (though, if you have the creative talent to make one up on the fly, a slight twist to it would be nice), as long as it adds a semi-plausible scenario for the PCs to explore. And they should be rewarded. Whether it is the same or equal reward you had planned for in your pre-made adventure, or less to encourage them to stay on track in the future, is up to you. I would, however, highly discourage the rewards to be more, unless you want them to make a habit of screwing over your plans. Don't be too lazy, though. The people and places the PCs encounter on this spontaneous adventure may well show up again in a future planned adventure.
Keeping these principles in mind, added with a little imagination, can make for a very fun night at the gaming table. Sure, there is nothing like the well oiled machine that is a thoroughly planned adventure, but there is a certain charm to one made from thin air. There is an unexpected quality to many of the elements, as often even the DM doesn't know what's going to happen, or who's going to show up, until the event occurs. There is always a difference in tone and feel to a spontaneous adventure, and it is a good time for PCs to really let loose, test you as a DM, and generally have a lot of fun. It's a change of pace, and a welcome one, generally speaking. You, as DM, shouldn't make a habit of it, but you shouldn't rail against it, either. It's a really good opportunity to experience something new, and maybe even refill your inspiration pool for future adventured.