Depends. There's alot of different kinds of fishing going on, so if you looked like you were shooting docks for crappie and a bass fisherman pulled in ahead of you and proceeded to work down the bank towards a point then I can't get upset at the bass fishermen.
A week ago some catfishermen were anchored on a shallow flat and I worked over the logs pretty close to them. Close enough that we had a short conversation at normal volume. Nobody got uptight. But if a guy had been out on that flat obviously bass fishing it I wouldn't have pulled in on him.
You can usually tell what kind of fishing someone is doing, and if the kind of fishing YOU are doing isn't going to effect them then go for it. If I am bass fishing and a crappie guy pulls in ahead of me to shoot a dock that's fine with me. I've picked up some good clues/tips by having another fisherman within sight. Sometimes you see them catch one doing something that you hadn't considered......so next time instead of getting all worked up, pay attention, keep an eye on them and you might learn something.