Charmed mobs can always be released, this should not be a problem because it takes only a few more seconds to complete the dungeon. It saves Player-SP, it contributes mob-damage and/or one or the other mob-buff to the party. The only annoying thing is, that Healer Hirelings would heal charmed mobs to full HP, wasting their SP on them. This is the only real annoying thing, maybe melees get a bit annoyed too when their attacks come to nothing. But this is a low price compaired to the advantages.
I thought a bit further about splashing Druid and there seem to be a few viable builds when Wizard 17 is the minimum to get the Mass Hold Monster spell. Splashing Rogue additionally for Trap skills only or +Evasion should work too.
1. Wizard 19 / Druid 1 (pure Enchanter, more high level Wizard spells)
2. Wizard 18 / Druid 1 / Rogue 1 (Trap skills)
3. Wizard 17 / Druid 3 (+1 DC for heightened Charm Animal; Animal Trance, Hold Animal and Cure Light Wounds (Tier 2 Druid spells))
4. Wizard 17 / Druid 1 / Rogue 2 (Evasion)
I'm not sure now which path I'll follow. #3 and #4 appeal to me the most, but on the other side more high level Wiz spells and longer duration of Wiz buffs are also good things to think about.