Click the little green icon next to each party member (or hover, whatever!) and a volume slider appears and lets you turn that one person down or up as needed.

Oh and a "Audio chat master volume" slider to go with the other 7 or 8 less important volume sliders. As it stands I have almost everything turned down to near zero and still wish I could turn voice up.