Maybe they don't understand what you are up to.
If someone fits in, maybe in a weak way, but still assimilated, people can understand his behaviour. They know how to act upon you and what is expected from you.
So if someone behaves strange, the person could be dangerous, unpredictable, unable to be helped, etc.
I don't judge you, I know the taste of life you are describing. Just trying to understand people's behaviour and share my thughts and insights.
Another point of view might be, people stop when you make them feel bad.
So they might be empathatic, they are able to feel into their surroundings, and willing to help. Helping feels good.
But if someone or something makes them feel bad, they will avoid it. Just as everyone, people avoid pain.
Status, or at least being "part of the group", definitly comes into play. I remember a social experiment years ago, in which a guy in a suit pretended to fall in public. Everybody rushed to his help.
Guess how the situation went, as someone looking like a homeless fell in public. Noone stoped by to help.
Maybe it's the fear of being associated with an outcast. Maybe they are looking down on him. Maybe they are disgusted. Maybe they are affraid.
In the same way, people might even look up to someone waering a suit, wanting more to be like him, or seeing him as "equal". And of course, everyone will see how helpful you are to
others, to those you are not viewed controvercially.