I don't fish much in the Downtown GB area but I can imagine there are a lot of onlookers and rock throwers that will probably keep me in other places. Not everyone is annoying that hangs around, but usually there is one bad apple in the crowd.
Or there can be tourists out on the rocks before you get there and if they were messing around it may put the trout down for a while.
Thank God there are thousands of miles of stream so close to there. Try the Greenbriar area if you don't want to go too far up in the woods.