WotC had made the same mistake as TSR during 4e. Between 2008 and 2012 they put out a book almost every 5 weeks like clockwork. And then were surprised when only the "core" of PHB 1-3, DMG 1+2 and MM 1-3 were selling decently while all the setting, world building, power source and adventure books were selling like garbage.
it's actually worse now because you forget two things: dungeon masters guild and official WOTC stuff half the time being utter shit. now guess who gets a nice and juicy cut for all the necessary "fixes" and "plugins" sold on dmsguild? authors get a 50% cut, up to you to estimate how much WOTC takes of the other half.
(I also don't think they released any 4e content)
hardly anyone did due to the SDL and WOTC wanting a cut of every sale, while preventing another paizo situation. since that obviously didn't work out as planned (and was retarded to begin with), they gone back to the OGL and reap in money from third party sales and get their brand strengthened by everyone selling a "5e" product.
For example, Return of the Runelords, there is an option to keep Xanderghul alive, by accepting his surrender. Personally, I'd take that route, because one of the most powerful ancient runelords, one that manages to become a demigod, alive to meddle in the present day seems far more interesting than that fat jealous hag Belimarius. The very next chapter, however, assumes outright you kill Xanderghul and you somehow will never, ever kill Belimarius. So in the end, the two Runelords that survive are the two surviving women. One is a tyrannical seductress-sorceress with insanely high CHA score, and yet adventurers are assumed to instantly trust her instead of asking whether she is doing some charm-thing on them. The other one... ah who knows.
The Lost Omens 2e lore all follow this. Whatever decisions you made in a certain AP that don't agree with what these people want to be canon - who cares, negated by lore.
I still take that over "what, you mean that arc has an ending? all the plot hooks lead somewhere? nah just make it up ;^)" in current official 5e material. look up rime of the frost maiden to see what I mean. avernus is apparently even worse, but I never read it myself. and the less said about radiant citadel the better.
the argument is also a bit disingenuous. if you buy an AP, you want to be railroaded, more or less. and fuck me, most people playing PNP these days, 5e especially, WANT that shit, even if they claim they don't (as if they would ever know the difference between a railroad pulled of in a smart way and "true" sandbox). and even if not, nothing prevents you from treating it like any other sourcebook ripping out bits and pieces for your own shit. no one complains if you throw out forgotten realms lore or whatever, but suddenly that seems to never be allowed or even possible when it comes to pathfinder stuff (I'm aware that makes me sounds like a butthurt paizo apologist, it simply reeks like the same "you can't RP in dnd4" shit from 10 years ago).
it's also a fact meta-plot advancement is what the majority wants these days (there's a reason even fucking warhammer does it now), and in a micro scale with their own stuff. they don't want to make their OWN adventure, they want to be part of the BIG THING happening everybody else knows about. they can't talk about random stuff with their other "nerd" friends if they never heard of it or can't grasp it - same way you can tell almost anyone about the crazy shit happening in your skyrim playthrough, but try it with dwarf fortress and see their eyes glaze over. that's just how normies work, and again 5e especially (and to an extend pathfinder) is fucking full of them.
TLDR: "canon" exists because people demand it, and I rather have it and can decide to not use it, than pay money for shit that's borderline unusable (and certainly is in it's intended function) anyway.
Am sure someone will beat Paizo to the punch, there are already a few projects like Advanced 5e far into development, not to mention the amount of people doing compendiums of their house rules, homebrews and alternative classes.
they've been done for a while:
https://www.levelup5e.com/
there's lots of other stuff around these days to fix the (perceived) issues of 5e in any shape or form. sure 6e will sell due to the official brand name alone, but anyone who already jumped over to an alternative has less reason to come back whatever WOTC is gonna call it.
"Black" was in there somewhere so that's probably it.
Some neat concepts but simply far, far too German for me to get into.
left: DSA player learns D&D
right: D&D player learns DSA