I've rarely seen people be negative towards fat lolita dressers, they don't look as nice but fats are fairly common. As long as you're otherwise put together it's not that big a deal, you won't be as popular as your thinner counterparts though.
That said the giant barrel chest of a man and his 5'o clock shadow are not fat or unshaven. They're man attibutes.
Lolita fashion is incredibly expensive so the gatekeeping for grooming to wear it exists because people don't want their photoshoots and events ruined by trashy, cheap, and unkempt looking people. It's popular with girls whose intent is to mimic the look of Victorian porcelain dolls and it was / is already highly fetishsized. The last thing they want is some adult men in aliexpress cheap shit ruining their tea parties and photoshoots. Especially when these same troons are most commonly in the really fucked up sissy communities when they dress this way.
Nobody wants to deal with an older man who talks like a baby about inappropriate topics and is probably wearing a diaper getting his rocks off at your event making everyone else incredibly uncomfortable.