Because they all do it. Anthony is confirmed to have fucked a porn tranny (Sue "Shperehunter" Lightning) - and probably didn't start or stop there.
Norton, Patrice, Bobby, and most of the Cellar Crew would do sex tourism in various 3rd world countries and I'm sure Ladyboys and/or full trannies were a part of it (if not even grosser stuff). The only people who could body Norton for being a degenerate are people who are just that much further above him (like Colin, Burr, etc) and likely don't care about him on a personal level. It's also not socially acceptable to attack trans people openly.
The larger reality is that comedy is a rough job for relationships and is comprised of broken people. You spend a lot of time on the road and away from home and bored with cash - a prime target for hookers.