Whirlibird gave few good suggestions. I do not think your problems have too much with you being prepper or survivalist. You simply have nasty neighbors.
And you are probably right, when SHTF, they will pay you visit very soon.
But being “not interesting” is first layer, second is that you show them their attack will cost them a lot.
I would simply be ready (however wrong may sound) to eliminate them on quick and efficient way, when SHTF