Ah well, then they're simply ignorant, I had the same happen when I was playing Overwatch with my friends back in the day, we were a big group of about 5-6 friends playing together daily and I always left when this one girl joined and I wasn't doing it on purpose, but she thought that I was trying to avoid her. In these scenarios it's best to have a 1 on 1 talk with said person and let them know that it's not the way they think it is.
CTB is never the answer when it's over friends and etc, it's only reasonable when you've exhausted all your other options and there's nothing else left, please re-consider and just talk it out with your friends.