Here is a comprehensive description of actions which lead to reputation change, and by how much: http://meta.stackoverflow.com/questions/269653/why-did-i-gain-lose-reputation-can-i-audit-my-reputation-history

So: when you unaccept an answer, you get -2 and the answerer gets -15. In other words, you reverse what happened when you accepted. Same deal if you then accept someone else's: you get +2 and that answerer gets +15.