As
@EnemyStand said, what works for your group is most important.
Also I'd be very hesitant to change course mid-stream; for the rest of your campaign I'd just keep doing what you're doing. Maybe possibly tell players what you're carving out for house rules so they don't get an unpleasant surprise at other tables.
For myself, as a DM, I usually hold to tracking free-hands & inventory because if you don't it makes things like Quickdraw worthless to take as a feat, etc. In a big budget game like D&D these things are almost always factored into the balance. But it does depend on the party - if everyone is just playing their character and not trying to break the system, if speeds up combat I'll probably let it go; but giving everyone quickdraw as bonus feat would be a complete non-starter with powergaming munchkins.
For casters, depends on system. Specifically for D&D 3.5/PF1e, fuck casters - they can sob all they want about needing to have both hands free to cast all they want as they end entire enemy squads in round 1.
For grapple, again depends on system. Again for 3.5/PF1 I'm going probably require both hands be free; I have to get out the flow chart, they can make sure they put away their weapons. I forget how grapple goes for 5e, but I remember it is less complex so I'd probably require at least one free hand.
Also in just about any system I give players their main weapon + two "Belt slots" that take whatever the equivalent of a minor action to draw from, and they can 'holster' a weapon/potion in them for free. (if its a two handed weapon, its just across their back). While it takes a minor action, I'll usually let players break strict action economy to get items in them-but if anyone wants to rules lawyer action flow, I go back to RAW.
So for melee multi-attack, for 5e as long as they have their bonus available, I'd let them have a single weapon swap in the middle of multi-attack. A lot of OSR-type system have full-minute turns, and a minute would let you draw and sheath several weapons.
Additionally, I look a lot a themeing if a player wants to flex some rules. "Are they doing something that works with their character, or are they just fishing for a +1?" If I have a barbarian who worships moradin with a lighting axe & thunder hammer who shouts something about Moradin's judgement as he uses both to wail on an enemy and does that every time he rages and multiattacks regardless of enemy resistances, I'm going be more likely to let that slide vs the "Well let's see. I'm going to use my sonic hammer, and then because
this is an aberation it will give them a vulnerability 3 to attacks vs lightning, so for my next attack I'll use my lighting axe which I've never done before and only do because there is a numerical advantage to be gained".
Personally, as a player, I like having limiters. Its fun to have obstacles to over come, and when you can do everything yourself there's no need for teamwork. You never appreciate Quickdraw if you never had to burn actions to get your other sword out, you never appreciate the Fighter holding the line if your caster never has to roll concentration, etc.
What I try to avoid from both a GM/Player perspecitive is needless autistic bookkeeping.
