Trail difficulty is a matter of perspective. Sam's Creek is covered by one of the easiest and most cleared manways - but it's still a non maintained (officially) trail. Anything off trail is a matter of preparation and knowing the area. Without knowing you or your grandfather's physical health, level of activity I can't really say how difficult you would find it.
The fishing can be great, a search of the forum will give you lots of info on trip reports and what the area is like.
I had a great trip up there with Duckypaddler two weekends ago.