The main reason is the uncanny valley effect. It's the same reason why extremely realistic robots, realistic CGI (like the Polar Express), and robotic (i.e. Siri) voices are off-putting.
Even in passing and semi-passing MTFs/troons, there are still male characteristics. In the less feminine troons, you get to see female clothes, makeup, and mannerisms plastered onto a male body, which creates that feeling that "something isn't right."
We can better see the contrast when you compare troons to actual "outlier" women (i.e., tall women, square-faced women, women with deep voices, etc.). Actual tall women tend to be lean and lanky, while tall troons are built like tanks with broad shoulders. Actual square-faced women include Audrey Hepburn and Emily Deschanel, and they don't have chins the size of baseballs. Kathleen Turner has a baritone voice, and she still sounds a trillion times more feminine than Caitlyn Jenner.
There are many subtle differences between men and women, some of which have very little overlap. Most people are unaware of brow ridges as a sex determiner, but once you see them, you can never unsee them.
