My rules: Go fishing when I can and preferably when its nasty weather outside!!!
Its amazing how few people want to fish in the rain.
Of course, the fishing is definitely NOT always better, but at least the streams, rivers and lakes tend to be uncrowded...
Seriously some good info on this thread for people wanting to avoid the crowds. I remember hiking in to upper Deep Creek one summer and hiking downstream for a few miles to reach "remote" water that was presumably unfished. Well, each campsite had multiple tents and empty fly rods were leaning against trees all over the place it seemed. Lesson learned and included a walk back upstream for 2 miles before we reached unfished water.
There are lots of other good tricks that time on the water will teach you to catch fish even in heavily pressured or travelled areas. Deep nymphing (with the correct flies is one easy solution. Learn all the tricks and you can fish major park streams year round and still do well.