The bounty will not be awarded to a deleted answer and it will not be awarded retroactively. Thus, to temporarily delete the answer until the bounty ends indeed would be an option to avoid getting the bounty.
I do however not quite understand the rationalrationale. For the person that started the bounty the points are gone no matter what; if the bounty goes to no one the points are just lost. Thus, you do not "help" the OP in any way.
If it is just you feeling uneasy about your undeserved increase in "wealth," perhaps rather then throwing the "money" in the trash, donate it to "charity;" that is take the points and start a bounty on some deserving question yourself, it could be even that same question.
(Technical footnote: if you should get just half the bounty it does not suffice to start a bounty on its own.)