Honestly I wouldn't mind Fallout 4's base building as long has it was heavily contained. The base building was a fun time killer but the problem is that F4 treated it as a core pillar rather than a fun side activity. If you had like three Hearthfire-esque plots of land and/or a small village à la Helgen, where you could build up your own home/base, I don't think it would be a problem barring three points:
A) It shouldn't consume significant development time/resources and should be the first thing cut or pushed to be post-release if it is doing so.
B) It should be done in the style of F4 rather than Starfield because, bafflingly, it was somehow worse in Starfield despite it being a core part of the intended crafting loop and having several skills linked to it.
C) No major questlines should interact with it, at all. The Mages Guild should not say "well all you need to do now to advance is build a mage tower and place the Deadric McGuffin in a 8x8 room with a-". It should be it's own island of content with a minimum number of quests about the building and only about the building and should not infect the rest of the game.