Yeah, I wouldn't say that liking Japanese culture, language or even anime in and of itself makes you a Weeaboo. I don't speak Japanese, but I certainly think Japanese history is interesting and I do like several animes. The key difference is that I don't think Japanese culture is perfect, let alone that its some sort of futuristic anime paradise. I find lots of other cultures and places interesting too. Thailand is interesting, Lithuania is interesting, Brazil is interesting, Uganda is interesting... that doesn't mean I need to sperg out over it.
I think the best way to describe it is to say that Weeaboos fetishize Japan and Japanese culture, sometimes quite literally and blatantly. Quite a few seem enamored by the idea of Japanese women, either viewing them as meek, submissive playthings or as living proxies for their anime schoolgirl waifus. But it goes beyond that. They'll ramble on about how katanas are the best weapon in the world, able to cut through anything. They'll dismiss or even try to justify Japanese warcrimes during WWII. They'll talk about Tokyo as a wealthy, futuristic paradise where overweight basement dwellers like them would be perfectly at home without realizing that Japanese culture has its own set of norms and (yes, unfortunately) prejudices.
Being a normal person who is interested in Japan is different from being a Weeb. It's just hard to see because Weeaboos are so prominent. There are far, far fewer people who sperg out over, say Spanish or Namibian culture, than there are Weeaboos. Unfortunately that means many people who just have a casual interest in anime feel guilty mentioning it because they don't want to be seen as obsessive spergs.