Do not worry yourself too much, especially over something that you cannot control. It is obvious that you cannot control who gives you + or - Karma. However, like every good and improving system, there are methods in place to check Karma abuse, so if it becomes a recurring issue, the admin can step in, just enjoy the forum.