I'm really sorry to hear that. Most people are pack animals who love to attack people who don't act like they do. For some reason a lot of people are really motivated by discrimination. If they can discriminate against some people, they will discriminate against some people to make themselves feel better. They love to think the worst of others. "That person doesn't wear a mask? Maybe they forgot? Maybe they don't have money to buy masks or they ran out of masks? Maybe they have a medical condition? Maybe it's really cumbersome to use a mask with glasses? Maybe they're a jerk?". A lot of people will choose the worst option, because of course they will. Made of course more ironic after reading some studies that say that masks either don't help or help only a little bit.
Is there any kind of a badge you could wear to signal other people that you have a medical condition and because of that you can't wear a mask? Doesn't have to be official. Even a home-made piece of paper taped on your coat or something. A note/badge saying "Because of a medical condition, I can't wear a mask." You may think it may look really stupid or embarrassing, but at least if I saw someone having a note/badge like that, I'd be a lot more understanding.
Another option would be wearing one of those really big scarves or scarf tubes which go so high up they cover your nose and mouth, so people can't see that you're not wearing a mask - or at least most people won't.
Ah, and of course those "lumberjack masks"! I have seen a lot of cashiers and some customers wear them! I don't know how to say this in English, but I mean those see-through masks lumberjacks and such wear to prevent pieces of wood flying into their face. Made of plastic. At least in my town lots of women and men wear them. They don't cover your nose and mouth so you can breathe freely (and use eyeglasses), but at the same time there's a protective "wall" between you and other people so your spit/sneezing won't hit others and the other way around. I googled, I think they're called face mask visors or face shields in English.
In Finland at least, there are "self-help" cash desks/registers/checkouts (I don't know how to call them in English). Meaning you can just quickly run into a shop, take some food and run to the cash machines and pay your groceries without interacting with other people or cashiers or waiting in lines. Combine that with a big tube scarf and not even the cashier will know you're not wearing a mask. You could use them, if they exist in your area.
The last option is of course that you only put on a mask when you enter the shop, and remove it immediately afterwards. Or try a different type of mask. I have used some really good masks that make breathing really easy, and some bad masks that almost made my mom faint because she couldn't breathe (has asthma).
Your government really should remind people that not everyone can wear a mask. In Finland, there were big newspaper articles and interviews of people who can't wear a mask because of a medical condition. Your country should do the same.
I'm sorry you can't see your grandpa. I hope corona goes away quickly so you can meet again. And those quarantines suck. If only you could take a test, and if it was negative, you could end the quarantine, so you'd only be in quarantine for let's say five days instead of two weeks. In Finland they reduced those quarantine times from 14 days to 10 when they realized most people won't be able or willing to stay at home for so long (and no one is supervising us). I was in a quarantine for nearly two weeks (not mandatory, but I was sick and it took a long time to get corona test results (wasn't corona)). I went crazy staring at the white walls. Thank god, VR exists so I could travel while being at home. I would ask if you could use Skype or other video calls to chat, even if the answer is probably no. I'd suggest at least asking him to take photos and send them to you and you could do the same to him. Or even videos. I really missed my dog during those two weeks, but seeing new photos and videos of her cheered me up.
Governments usually suck, yeah. Can you ask people to bring non-fruits/veggies? Like, ready meals? That way, if you're worried about them touching the food, they won't be able to touch it because it's inside a plastic/carton packet. In Finland, there are ready meals like spinach soup, tomato-basilica soup, several kinds of porridge, smashed potatoes with anything, tortellinis. I hope those exist in your country too. Some of them are fresh from oven, so you don't even have to prepare them. And some of them you just put in the micro and you have a meal ready. And for example carrots, in my shops, you can either hand pick them or buy in a plastic bag so no one has been able to cough at them or touch them so you can be sure they're clean.
I read yesterday that magnesium and D vitamin help against panic attacks and depression plus lower the risk of them. You don't have to get them from food, you can use vitamin supplements.
I hope things will become easier and better for you! And that corona will go away soon.