Deleting it does, indeed, erase those lost votes and you get your points back.
Deleting answers that are deemed low quality by the community is also desirable in its own right. It helps keep the site clean from clutter. I do encourage you to go ahead and delete it, but if you want help in writing answers more effectively, and that will get up-votes instead of down, I have a few links for you.
