Elkmont is a fairly crowded campsite. If you have gear with you for backcountry, you should park at Elkmont (at the end of the road) and hike in. I've been up to #24 twice now and that trail is a converted railroad grade and is a super easy hike. Some sites (#24 included) require reservations so make sure you reserve a spot.
If you do frontcountry camp, it's pretty easy to get away from the crowds. Take "shanks mare" as Mr. Casada says and just walk a couple of miles in. It amazes me that folks fish right on top of each other when 15 minutes of easy walking would put them in water that's much less crowded and overfished.