Season 4 pacing has been perfect imo. One more episode of season finale and there's been a lot of plot / reveal all season and the interaction with the so far main antagonist Man in yellow.
I will say part of me thinks the man in yellow is also just a player in the game with what's been said. His observance of various rituals and with him being sophia he's not some omnipresent figure. He doesn't know what's happening unless people tell him and on some level he has to make sacrifices for his 'magic' or whatever to work. I've also never seen him really interact with the nightmare face suckers cause he's only been seen eating people during the daytime.