This is a very good idea.
Since the question was asked there has not really been any progress in making the peer-review system healthier. Also initiatives such as Plan-S in Europe have not really contributed in this direction. There seem to be larger and larger masses of papers submitted to classical peer-reviewed journals and most of them are congested.
My opinion is that such a system (posting abstracts of linked arxiv articles on MO) and asking for reviews would be a big relief to the system. However I think that the main problem would be of finding a way to include anonymized (sponsored) reviews (which is a very important aspect of the peer-review system). Maybe one could have a group of moderators acting as editors and they would be responsible for searching/receiving a review (which could also come from outside the MO community) and posting it as an answer.
I also think that math annotations and reviews are two different things. I would first stick to the review context, where answers are only allowed to be classical reviews and comments would only concern specific (technical) questions. This would imply some sort of "strong" moderation from the community, but I think that the success of MO has shown that this is possible (I myself would be prepared to be involved in sections related to my subjects - for sectioning we could for example use MSC classifications).
Moreover, I do not think that MO rep of the user (author) who submits a paper for review is going to be relevant in the hiring process (or at least as relevant as RG's score or H-index etc. is), but I somehow envision this MO section as becoming a new type of "journal", which assigns reputation to an article. Having a linked paper with an "accepted review" and a good number of upvotes could be considered as having a paper with a substantial impact.
If you ask me, I think we should just do it and see how it evolves. Otherwise discussions will just continue until the people don't have anymore time to invest.
EDIT: I have posted an example review here (Example) Peer-Review request: Independent, Negative, Canonically Turing Arrows of Equations and Problems in Applied Formal PDE
Hopefully I have not broken around 1000 forum rules.
EDIT2: I have deleted the above post and will give a try with Gerhard Paseman's article he was kind enough to "volunteer it" for a community review. Since I regard the current problems of the peer-reviewing system as dire, it would be great if the downvoters also comment on the subject. The main goal is to find a way to improve the system-this is a problem of the community and it has to be solved within it.
EDIT3: Gerhard Paseman's article has been posted: Peer-Review request: Updating an upper bound of Erik Westzynthius