I've noticed that I lose 1 rep for every downvote that I do, even if I leave a comment to explain. Is this supposed to be happening? I thought that the rep penalty was only for if you downvote without leaving a comment?
It's not commenting that makes a difference, but the type of content that you are downvoting. Downvoting answers always costs you 1 rep. Downvoting questions doesn't reduce your rep at all.
This help center page covers this and more: What is reputation? How do I earn (and lose) it?