There’s so much to talk about here. First, when you’ve got spots that are that good, the fish will be stacked up and it’s so difficult to put your bait in front of a 20 incher’s mouth before those 16-18” fish get there. There’s the competition thing going on in full force.
As far as casting to the “heart of the the spot” first, I tend to agree with your fly fishing buddy. If indeed the big one is in there, there’s a better chance he will get it before you hook a 16” fish and alert the whole school. Then they will all know Uncle Al is back in town for his annual float.