Sounds like you tried several files and had them interested. That's a pretty good sign.
You saw them = they saw you. Your odds just went waaay down. Cast to where they ought to be without seeing them for sure. Also, remember where your rod tip and line are.
It could be something was off in your presentation too. You need to get a natural drift and not have a lot of tippet or line on the water. Also, you need to have a really light landing with fly, tippet and line.
4 pound is too heavy. I'd go down to a 6 or 7x and longer tippet if they were acting that way, or if I was casting into unbroken water on a tiny creek like that.
There's all kinda little teeny stuff in the water that they eat and you can't see. Some of it is right in the film, and they'll come up and slurp it. In that case I'll sometimes toss an indicator fly (one I can see well) with a teeny (20-24) emerger tied about 10 inches behind on 7x. Just a little tan or brown dubbing on a hook.
Also --- take it slow. If you're getting short takes, let it rest. Give it a good 5 minutes -- until it's painful -- then try again.