The reputation tab on your profile will detail all the changes to your total. Viewing the timeline rather that just the graph can be helpful sorting out what happened. For example the time slot you gave (depending on what time zone you gave it in) might be explained by this bit:

That's somebody removing an upvote on your question after you edited it (-5) plus two positive entries of +2 each making for the overall -1 you see in your total.
If that's the wrong time slot, I suggest you might be forgetting to factor in the -1 rep when you downvote somebody elses answer which are NOT shown on that timeline to protect your privacy (what posts you up or down-vote is not indicated anywhere).
Negative changes typically don't trigger new notifications, although recieved downvotes do appear in the rep activity list and you can see what posts they were on if you click on those events. On the other hand the rep you've given away in bounties seems to be the major down-hits for you with a few minor ones being downvote related. There is also one case of a post being migrated to another site and another deletion or two. Deletion of posts (at least recent ones) will cancel any rep changes they originally provided. I don't see anything out of the ordinary with your profile other than these routine fluctuations.