Middle Prong can be a real tough stream to catch fish in the park. The fish are not as opportunistic as other places in the park. There are definitely easier streams to catch fish on, so don't get too frustrated. As you noted (and like many things) perseverance, practice and some hard work (hiking in) will pay dividends. The fish along the road see lots of people, but only the better anglers can catch fish there with much consistency.
An example for you from another current thread. During the fish-out on Lynn Camp (a tributary of Middle Prong) a few weeks back, two anglers caught almost half of the fish that were taken during the first 8 days of the event, even though over 100 people tried their luck! Most folks caught 0-2 fish...experience matters
Like you said, the fish are there...
"Even a fish wouldn't get into trouble if he kept his mouth shut."