Onehot if you are sitting a worm on the bottom, in that situation the attraction of the worm bait may override their caution and line size may not matter as much like TJM mentioned in his post. You may actually catch more trout if you dropped your line size, but you are not trying for them anyways.
I have seen similar responses to what TJM mentioned in clear water and mostly where the fish have a long time to evaluate the bait. In those situations I will almost always go to 2# line. In situations, like jig fishing a river like the White, 11 pt, etc., those fish are reacting quickly to a bait rushing past them. I don't believe that they are as line shy. At those times I prefer 6# fluorocarbon leaders.