Believe it or not, there are actually players who don't care about wringing every last possible drop of XP out of the game, because it simply doesn't make any difference to how they play. If you added up all of the XP I've lost out on due to not having my voice in place at the ends of quests, it probably amounts to less than an extra hour spent in the Vale at level 18.
I'd rather spend dozens of hours with my character properly geared.
(I rarely use XP pots either, and never buy boxes or whatnot. Wow, it's a miracle I even know how to breathe!)