I have tried several antidepressants for myself before I "presented" more accurately to my psychiatrist how I'm really bipolar and not depressive. He figured it out in a session, because I was manic.
I tried and quit several antidepressants cold turkey, on my own, without advising my psychiatrist that I was quitting my meds. Thought why pay him $300 just to agree with my decision that I decided to quit? (Learned this was a mistake for myself).
I quit cold turkey with Lexapro and Celexa. For these two, I quit no problem. Then, I tried the same approach with Paxil. It was a bad mistake for me. Had weird side effects called "brain zaps", per my psychiatrist. Apparently, this can be a common side effect if one does not tapper off these medications properly.
I've read the longer one is on antidepressants, the more likely one is to have side effects when quitting.
Fyi, I was on none of these three medications for longer than six months before trying to quit.
Be careful how you decide to proceed. Recommend you work closely with your doctor/psychiatrist to minimize any chance of getting brain zaps or having any other side effects.
As an aside, I agree about potential long term side effects that you mentioned. All these meds are nasty!
For my bipolar, I take the atypical antipsychotic Zyprexa. Luckily, I don't have to use it daily - only as needed for manic episodes. Zyprexa has one of the most nasty side effect profiles of any of these medications. I wouldn't take it otherwise, but I tried about 15 medications for my bipolar and Zyprexa is the only that works for me; and, for me it works great.
Good luck.