Ironically, anime has been more acceptable outside of Japan for normal adults than in Japan, from what I've gathered. Manga is a different story, as you'll soon see.
According to the Wikipedia article on "otaku", the stigma is dying down, so it would seem, and liking anime in Japan as an adult is becoming less odd. Also, the Japanese people don't automatically think otakus are pedos, but they have often thought of them as really undeseriably weird. An example of how otakus have been viewed in Japan can be seen in the first volume of "Oh My Goddess!". tl;dr: fat asocial unhygenic guy in glasses who spends all his time obsesively watching anime intended for kids and tries to molest Belldandy.
I've posted this before, but I've talked to a guy from Osaka awhile back when he was visiting my country. According to him, anime was a kids or geek thing ("geek" was how he translated "otaku" into English), while everyone reads manga*, and vidya (aside from eroge, that is) was acceptable for adults according to him as well. The impression I got from him was that he really disliked otakus.
*Manga sales are about 40% of all book sales in Japan, so he's at least right about manga.