People like this thrive on bringing others down in order to make themselves feel important- showing that you are upset by his comments gives him fuel to continue.
I know it's hard and easier said than done, as someone else said, but the key is to try to let it slide off your back - his comments are down to his own insecurity rather than anything to do with you!
Feel pity for this pathetic soul who needs to drag at others in order to make himself feel good - his reliance on this tactic is a testament to his own weakness, not yours.
If you refuse to give him the reaction he seeks he may well double down and make more of a scene - this will serve to make him seem even more like an idiot and he'll show himself up sooner or later. This is something you can openly laugh at - why should he be so bothered by anything you do if you don't appear bothered by him? Fake it at first by pretending not to care; hell, call him "Karen" for interfering with your business uninvited, his opinion means nothing to you and his input is meaningless.