The last time I remember being even remotely flattered to have my appearance pointed out was around age 11, maybe? Being called beautiful by family friends whenever they'd come over for dinner or getting notes passed to me in class. Innocent stuff. Then came the cat-calling (12), brazenly pervy teachers (14), waiting for the bus in the middle of winter, and being approached by an old guy asking me to follow him to a public bathroom and give him a blowjob for money (15) and so on. Getting into fights with girls I'd been friends with for years because someone's ex-boyfriend mentioned me to someone. I think this and other similar things turned me against the idea of human contact altogether. Nowadays I just want to be left alone and I find even sweet and kind confessions appaling (though I'd never show it). So, yeah, troublesome in general.
People who aren't attractive desire good looks because they think it's going to grant them instant happiness. Same way others desire money, or intelligence, etc. And those people definitely have a point. But it's not a golden ticket.
That definitely seems like more of a dilemma. Like Omelas or the trolley problem. Lives being improved and saved at the suffering of one person... I'm sincerely sorry you have to carry that burden.