Originally I didn't, it wasn't my intent, but they do fit better if I am bulked up with long johns and a fleece jacket.
I actually only wear them when it's raining or snowing. The matching overcoat is sized the same.....and fits the same. (XL in my case).
You are a bit taller than me, but otherwise pretty much the same size (I think).