Think Cosby Creek when you get there...a lot of hikers, particularly on the weekends, and the Colorado Trail passes right through the trailhead parking lot. However, much like Cosby, it seems that none of them are there to fish, so you'll probably have the stream to yourself. We didn't see anyone else fishing the lake, either, although it was in the middle of the week. I would suspect that, given that we're into September now, you won't have any problems with crowds.
If you really want to get away from everyone, I would recommend the San Juans, but that's probably a bit of a drive for you.