What about gear? A new player playing a Cleric takes a level of fighter and replaces his old weapon with a great axe. Roll him back to pure cleric and he suddenly isn't even proficient in his weapon.
I would be completely stunned if people lost levels with this change. It's just so much easier to give them the minimum for this level they currently have.