From the understanding that the criteria a user votes +/- on answer is that the answer was useful or not, what would it take to ensure that downvoter leave a comment before their vote is registered?
Could having them leave a comment anonymously, if that is what is preventing them from leaving a comment, help?
Another suggestion is that, in the event of only negative votes, and if the answer has been properly supported, and if the user who answered flagged the mods, if the mods may maintain the voting at '0' if there appears no justifying reason for the negative votes.
Not giving up! Seeking solutions!
3 Give a value of -0.5 to negative vote w/no comments and maintain -1 vote w/comment? i.e incentivize leaving a comment.
Comments especially for a downvote do provide the feedback required so that modifications can be made to enhance the answer.