I also hate traveling to fish...driving past perfectly good fishing spots!
But I do like fish variety, so...
As long as the travel costs weren't prohibitive (and I had plenty of vacation days available, or was retired), there aren't many places I wouldn't go, for certain species. Such as:
Alaska for salmon, char, lake trout, grayling, etc.
Texas for Rio Grand Cichlid
Louisiana for Black Drum, Sheepshead, Alligator Gar.
Florida for snakehead, the other bowfin species, barramundi, spotted sunfish, tarpon, florida gar, mud sunfish, redfin pickerel, etc.
Indian Ocean destinations for Napolean (Humphead) Wrasse, milkfish...
Australia for Murray Cod.
Brazil for Arapaima, Arrowana, and lots of other jungle fish.
New Mexico for Tiger Musky.
California for Sacramento Perch.
and the list goes on...