Certain spells require a weapons change, and depending on the content, could be frequent (i.e., Dispelga)
TBH, sometimes I lock my weapon because I'm lazy and forget to change weapons/Gem for potency. It never made any difference to me in 8/9boss Sortie, which is arguably where you would want the swaps for additional Magic Accuracy and potency on bosses (Distract and Gravity). Aita still moves slow enough to handle with Gravity II without swapping to regal Gem, and your magic accuracy should be high enough to land it. Also, most ilvl weapons have Magic Accuracy and high skill on it anyways, so it's not like you're lacking MACC to land anything.
Dia 3 does not require a swap. Neither does any upstairs boss Sortie Boss. Even still, disabling ammo and unlocking weapon still keeps your Hoxne Ampulla buff intact and only causes a TP loss (which is nothing with Hoxne+Temper II). Where else are you using RDM that
requires a weapon swap? There is no high end magically evasive monster that
requires such a standard. There is no buff (whether on yourself or others) that require the weapon or ammo swap. it's merely a duration/potency improvement that you will live without, I promise.
For cures, I like to switch to the appropriate club for 30% potency
When do you Cure on RDM, and why can't you make a 50% build without the weapon slot? Unless you're main healing, I'm not seeing the reason for swapping weapons as RDM while curing in modern content (Limbus/Sortie). Even when Main Healing, I prefer R30 Sakpata's Sword+Sacro Bulwark locked for the tankiness/idle Refresh.
I think you're making things way too complicated. RDM is busy, but you don't need to impose arbitrary rules and gearing standards for every spell/action. That just makes it overwhelming and unfun. It's no wonder you're struggling to figure out RDM, you're overthinking basic mechanics.
You even answered your own problem at the end:
So...it's a weird juggling act I'm trying to wrap my head around. Again, my assumption is that I'm just not managing it right...like I'm unnecessarily feeling like I have to swap.