None of us are on your property and have all the information you have with your interaction with this guy. I will say that I empathize with your situation. It took 5 years of vigilance, reporting, and prosecution. We got great help from law enforcement. Part of that was due to how we handled the situation. We had poachers kill young bucks and hang the carcass on our fence to show us who was in charge. ATVers thought they owned the place because a pipeline bisects it and the previous timber company ignored the trespass. Many encouraged us to retaliate (boards with nails to puncture tires, wire across trails, ...). We never did any of that. Instead we were courteous. When we would see a trespasser, we would go over to them and "collect information". "Hi, I'm jack, what's your name. Live around here? ...." Eventually I'd take a cell phone pic and ask for ID. We are set up as an LLC. I'd say things like "I work for the company that owns this and I have to check your ID. If you won't show it to me, they make me call the sheriff's office. If you do, we can handle this without calling them.
In Virginia, they have something called a trespass notice. You go to the sheriff's office and fill out of form the the name and address (thus the ID) of the trespasser. I think there is a small cost if you are a private individual, but they do it for free for an LLC or other form of company. A deputy physically serves the trespass notice on the named individual. Once served, they have formal legal notice not to trespass on your land. There is no "I didn't see the sign" excuse. If you see them on your property again, you simply call the sheriff's office and tell them. The trespass notice automatically becomes an arrest warrant. There is not discussion, they are arrested and a judge sorts it out from there. For first time trespass offenders, most judges are pretty lenient. Sometimes a warning or small fine. If there is a trespass notice in place, judges take it much more seriously. Fines are much higher and it can even carry jail time.
My point in all of this is simple. If you let your irritation drive you, it is very tempting to stoop to their level and retaliate. At that point, law enforcement views it as bad-guy vs bad-guy. On the other hand, if you come in with obviously clean hands, they seem to go out of their way to help.
If the guy is putting stands on the line, he is still legal. If you have a chat with the game warden (during the off-season when they have time) and express your concerns. He may be willing to help out and monitor the situation during the season when he has time. It is also good to acknowledge the rights of the other guy when chatting with law enforcement. They see you as more reasonable and balanced. For example, when bow hunting, I often hang a treestand on the back side of a tree from where I expect deer to approach. This allows me to sit and when I hear deer behind me stand using the tree as cover. I can also stay seated or stand and wait for the deer to walk underneath and passed me before I shoot. I'd say about 70% of the shots are behind the tree. I'm not saying that is what this guy is doing. I'm simply saying that acknowledging there are reasons why someone might tells law enforcement that you are fine with whatever he does as long as it is not on your land.
No two situations and you know yours best. If you are in VA, a trespass notice might be in order. I find that just the fact that a deputy shows up at the guys house and serves him lets him know you are serious and it increases the consequences if he gets caught trespassing. For many folks that is enough deterrence.
Best of Luck!
Jack