Younger Tom and I did it as a day hike two years ago. It was tough, but not so bad that I wouldn't do it again. Maybe it was because we didn't get to fish much and I've still got the itch for the stream.
There were 2 or 3 cars in the pull off. We checked them out by peeking in the windows and looking for bumper and window stickers. We convinced ourselves they were backpackers. Wrong! We got to the campsite just as three guys were finishing up their breakfast and pulling on their fishing gear. They were there first, so they got dibs on fishing uptream. They were nice enough to try and walk for half an hour before starting to fish. But since they had to walk thru the water in places, that didn't help much. Younger Tom started off downsteam and seem to have had about as much luck as you did Craig. I went a bit up stream and kept seeing wet boot prints and we both decided to bail out. We packed up, hiked out, and enjoyed a good day on Straight Fork (which by the way, will be closing November 12).