So after installing the latest beta AMD driver 13.6, without CCC, and still having the same problem, I tried one more thing. I quit steam and xfire. (remember portal also not working well, that got me thinking...)
I actually remember now some Macintosh programmer telling me about how steam on the mac runs. He said it uses tons of processing power just monitoring where the mouse is and what it is doing, even when it's not the foreground application and really has no business polling mouse location.
And that fixed the problem. So either steam or xfire was causing the problem, or maybe both.
So for those having trouble, try deactivating steam/xfire. It worked for me. (it may be a driver and steam interaction issue, but for now, I'll go with what works and stop fiddling)