The way I see it is you need to be selective on how you're selfish, living a completely selfless life is just self-destruction. It's like first aid - prioritize yourself before you help others, I think it also applies to suicide.
You can either spin it as selfish for wanting to leave or selfish for making you stay. If death is the best thing for you it is selfish to make you stay, it's similar to putting down a pet - tragic but it was necessary.
I dislike suicide being labelled as selfish as it is a personal decision and should be made independent of outside factors. To answer your question; No, you are not selfish for not caring, it is your decision after all and it's difficult to understand your reasons from an outside perspective. You control your own life, you know what's best.
Living for others isn't a permanent solution and can be pretty self destructive, you aren't a bad person if you CTB - I don't hold that decision against anyone I know who has gone through with it. You control yourself and life, don't obsess over what others want and feel - it's not their life to control.
I hope things can start looking up for you <3