I think it's because we're animals at the end of the day. Packs of animals will attack and even kill those that appear weaker, different or a threat. There are also power struggles within a group. A wannabe leader or bully will try to ostracize other members to get rid of competition, gain followers, seize control.
In human societies, we normally need to be taught to be compassionate to people who are disabled, look deformed or act strangely I think. I have this terrible memory of being quite young and we went to visit some friends of my Dad's. They had a relative there and I'm not even sure what he was afflicted with but he got so excited about playing the piano and I made some mean comment about it and was (rightly) pulled up on it. It wasn't really mean by the way. I definitely didn't call him anything. I think I was just a bit snarky- saying he'd gotten a bit over-excited. It was mean though. Very cleverly, the relative didn't exacty tell me off. She went in the other direction and said how grown up I was to know such a long word. It sank in though that it was a shitty thing to do because I still feel bad about it now.
I guess my point is, I don't think compassion towards someone different necessarily comes naturally to us. I wonder if we have to learn it through empathy. Through being teased ourselves. Some people maybe never learn it. Some people never really mature either. They probably think it makes them look strong/ superior to pick on other people. Plus, sometimes, disturbingly it works- other people laugh and join in. Sadly- we do seem to have brains that enjoy to hate 'the other'. Look at all the wars and genocides that have happened because one group of people decide they are superior to another.
The other thing is maybe it's a form of relief. We're a cruel species. We will laugh at the misfortunes of someone else- in part because we're probably just glad that wasn't us.
I suppose as well, if you unknowingly offend someone- they may well lash out back. They may not believe you did it unknowingly.