BB28, the only way to win on those swing-and-miss hits is to immediately stop the harness movement (or at least slow it down) and hope he comes back for the rest of that tasty morsel. The problem is that the fish has hit aggressively, bitten the crawler in half, and is no longer attached to your bait. Since you're drifting and not hand-holding, you'll need to immediately put your TM in reverse. Sometimes, they'll come back and hit again, even picking the harness up off the bottom.
Question: you are using a 2-hook rig, right? If you're not completely insane and fishing in gnarly timber like rps, you might try a 3-hook harness. Or, another trick Old John showed me for combatting short strikes: on a 2-hook harness, pin the crawler at its midpoint on the front hook, pin its head on the trailing hook, and let the tail run free. It shortens the profile without losing "meat" appeal, and actually simulates a crawdad to some extent. Works.
One last thing to try on short strikers--rps does this. Separate the spinner blade from the hooks--move it an inch or two up the line. You can take the beads up with it, leaving about a 2" gap, or move just the blade, or simply add more beads to get the flashing blade further away from the business end. Hopefully, that will make the fish strike the worm higher.