I think many wrong answers should be kept for the reason you stated.
But this only applies to answers that have mistakes that are interesting or easy to do; it is only worthwhile to keep answers if people can learn from them.
Comments and downvotes can indicate that the answer is not good, and ideally it should be also indicated in the body of the answer as well.
If an answer has a very negative score, the answerer may want to remove it to save their reputation (in any meaning of the word).
There are also answers that are not even wrong (to quote Pauli) — somehow nonsensical or unrelated to the question.
Those should be removed.
If the answer is not nonsensical, I think it would be better to let the answerer make the deletion call, as Gerald Edgar suggested in the comments.