Why is it so hard for me to stand up for myself?

You've been conditioned to avoid conflict and, as you said, to feel guilty if you engage in it. I would suggest that you talk to a counselor of some sort about finding some ertiveness training. That should help you to not feel guilty about standing up for yourself and to also stand up sooner so that the impact is less. Good luck.