.5mm can be too small but it might not be. You have to get some precision in the reduction before you can say for sure. And that PSI seems really high. I use a different brand of brush but it's rarely over 15lbs. You need to take some time and see how viscosity and air pressure affect what you are spraying. If you can get that dialed in then you can determine if .5 is enough.
The coatings I use are about the viscosity of water. Very thin so they spray well through it but I'm using a .8mm nozzle. And that's mostly because of the metallics and the clear.
At 50% reduction though you have it so transparent that you are trying to pile it on to get coverage. And probably not letting it flash long enough between coats. And at 40 PSI it's really blowing it through there. Reduce it 10%, cut back on the pressure, and see how it goes. Then you can adjust your pressure and your reduction to get something acceptable.