In short - yes. Yes you have to consider the possibility that you will receive a visit at some point in the future. The whole point is that nobody knows when a SN seller may get busted and what information they leave behind when they do. Nobody can offer guarantees about an unknown like that and if they think they can then they're deluding themselves or misleading you. So have in mind that it's a possibility.
Should you worry about it? Well, depends on your circumstances, location and what in particular worries you about it. I received one and it wasn't anything dramatic for me. Just a brief chat with an officer. The end. But there are different experiences, different actions taken, different living arrangements out there. You have to balance these things against your desire for SN, don't be crippled by fear.
.