I would let the early, skinny ones go. With two years in the ground you should be getting some good ones soon.
Asparagus is one of those things that makes spring exciting. I used to love having a little asparagus patch, but trees grew and there just wasn't the sunshine it needed. That's a problem with my whole yard, which is a bummer.