I, too hated the episode at first. And looking back, it was definitely an early example of one of the things that would go on to ruin the show- fucking with the characters, their back stories, and spitting in the faces of loyal viewers.
But, it grew on me in syndication. This is when I watched the syndicated episodes religiously every evening after school, later after work. Seeing it repeated, I came to appreciate the tension that builds through the episode. The mysterious way Chalmers follows Skinner around the school ("The rod up that man's butt must have a rod up it's butt!"), the shadowy figure that wants to attend the party for Skinner, the discomfort of the characters at the situation/"the real Seymour." The ending is stupid, but does kind of put a cap on it, and makes it so it need not be mentioned again. But I don't think it really takes away from, or really changes the characters. I daresay, it embiggens their back story. It adds some flavor to Seymour and Agnes's relationship, which had already been established to be weird and super dysfunctional.
Also, the episode gave us coffee-flavored beverine. I'll take it grey, with creamium.
I get the hate for it, but also think it's kind of a meme that it's "totally the episode that ended the show, dur hur."