To an extent I do this too -- within minutes to days of meeting someone, I've already sized them up and come to a decision about whether I'll "vibe" with them or not. I don't think it's based on appearance though; I couldn't tell you exactly what tips me off, it's just a gut feeling, though I'm sure appearance plays a role.
Regardless, I wouldn't consider it prejudice--which by definition happens before you interact with someone (hence the 'pre')--so much as discrimination, which in this context of discernment needn't be a negative thing; it is in fact an essential part of survival, and as @derpyderpins mentions, is something everyone does.
I would echo his sentiments that it only becomes "wrong" when you're too rigid in your initial conclusions and don't allow for human messiness; when "my instinct is that I can't unmask around this person" becomes "I will never be able to unmask around this person."