I think the software limitations are a big part of the problem. In particular, how comments are integrated with questions and answers. The comments appear to give an opportunity for people to have a conversation, but this is discouraged. Trying to have a conversation and getting shut down is not welcoming. The chat feature is not threaded, and so it useless for anything but synchronous communication. I think the site would be much more welcoming if instead of the current comment system, there was instead a threaded discussion forum associated with each question and each answer. In particular, I think people would be much more comfortable participating in the forum than the QA, but forum participation could serve as a gateway to QA activity.
Edit to include further clarification from a comment
I was thinking that there could be a "preview" of the top level "reddit" forum comments which would go where the comment section is now. So for the casual browser, it would look pretty similar to what we have now, the only difference being that only the most useful (upvoted) comments and replies to those comments would be visible from the main site. There could then be a "contribute to this discussion" link which takes you to the discussion forum for the question.