In the day...
I used to fish that complete stretch as well as the one above the Chimney Trail parking area a lot. Only on one occasion did I ever encounter anyone more than a few hundred yards up the creek. On that trip I ran into a couple of Indians from Cherokee. Each was fishing bait and carrying a plastic bread bag with what looked like a hundred or so little trout in them. They were not the least bit concerned with anything other than catching trout.
It's long haul on either stretch, but not one you cannot easy accomplish in a day. After mile or so you get into lightly fished water. It's about a 2, or may be 3 flask fishing trip, depending on the weather.