Only once so far, I was fishing on a Bella Vista lake, bluegill fishing, probably about a medium cast away from the bank - couple of yaks were working their way down the bank fishing, got close enough to start a conversation with us, friendly conversation, then they ask me if they can paddle between me and the bank, we were done for the day and I said sure, go ahead, probably would've asked them to go around if I had intended to stay. Thought it was kind of funny that they needed to go between me and the bank, could've very easily gone around the other side of the boat.
There's a gal on Table Rock that paddles down the bank as exercise early on summer mornings, I see her all the time, she'll go between me and the bank unless I'm really close to the bank. It doesn't bother me, if it spooks the fish it is only temporary.