I hear that.
I think one thing that brings me peace when I think about this is that ultimately, I know that my loved ones would never ask me to suffer for their benefit alone. I also try to remember that it's unrealistic to expect myself to be solely responsible for all the emotions and reactions of everybody around me, and that to try to do so is taking on a burden most can't carry for long.
I also remind myself of the fact that there will always be painful farewells ahead, whether you catch the bus early or stick around for a while longer yet. Your loved ones are going to have to let you go eventually - none of us are immortal. I can consider myself to be blessed to have formed such strong and good attachments, that they would be the only thing to give me pause on the way out the door, and still continue on my way knowing that all things will come to an end regardless of what I, or others, do and when it is done.
Don't forget to show, or tell them how much they mean to you before you go. Give them some good last memories now, so that they have a fresh, bright memory of you to warm them through their grief.