I think it's a mix of inherent personality and neurology as well as environment. Ability to experience empathy varies by person, but neuroplasticity can modulate it for a bit. But there is still a significant baseline for every individual whether they get to "practice" using that part of their brain or not.
Even a decent person may not be able to relate to the problems of those who have to deal with incredibly different lives. For example, those born into the upper class, uber wealthy are painfully unaware of the realities of the stress the lower class faces, from my encounters with them.
Many weren't even necessarily looking down on them, it was just something they'd never had to think about (partly because deep down they know it'd disadvantage them); Kind of like how the average person doesn't know the intricate details of the distress the animals we eat went through, just that we know they had it rough.
I don't think it's an inherent case of sadism for that many people, it's just everyone trying to survive in the only way they know: A species that is inherently codependent.