You have every right to utilize your property when and where you want so screw em if they get upset.
This same sentiment goes for the landowner on the other side of the fence too. If a person on other side of fence is hunting the opposite side ot the fence property, that is wrong all day long but if a person has a hunter in a stand on their side of their property, watching and hunting their side of the property, more power to them.
I own a couple small properties, 80 and 54. My stands are on the outside, looking inside and hunting inside. If I hunted from the inside my property to the outside, I would have no deer after the first sit or two and deer, unless dropped in spot would/could make it to the neighbors property. This would result in a recovery attempt onto neighbors property where they may be hunting or may be planning to hunt, possibly blowing up their property.
In the 17 years we've owned this property, we have only had one deer cross the property line after being shot, and it was in the first couple years when we hunted from inside to outside. It was a doe my wife double lunged that ran about 120 yards, dying 40 yards into the neighbor's land. Thankfully it was during the late antlerless season and didn't interrupt their hunting as they didn't hunt antlerless. After that season I changed tactics and hunting from outside in.
During this 17 years, an outfitter leased the 375 acres directly north and east of me for 12 years(starting after the year of the doe recovery mentioned above.. He had stands from 40 to 100 yards from my line(ones that I could see). During this time he asked me 4 times to recover deer on my property. He complained of my stands on my property, 5' to 20 yards on my property, saying he didn't like that I could see onto his leased land. He said I should have my stands in the middle of my property. I explained I would be pushing the deer of my property in a couple hunts and he said "that's bullshit". I really didn't care for him either as I caught his hunters doing a deer drive across my land and he refused to cooperate with the CO when I wanted to press trespassing charges.(caught via trail cam, non cellular}.
If one is hunting their property, what themdawgs said absolutely applies. Not all landowners are blessed with 100's of acres they can have stands all over and not blow out the deer.
The outfitter lost the lease last year and the new lease holder is a dream to work with and understands what our properties can do when working together. We have both taken the biggest bucks of our lives in the past two years. The potential has never been greater for both of us.
I also own my parent's 160 farm and I approach this the same way as above. I have a fence sitter on the neighboring property. He hunts his side and if a deer goes off my property onto his, he has every right to try to hunt it as long as he's got the tags and such. There is really no habitat for them on his propery except for rotating corn/soybean fields. My 160 is essentially all habitat via crp and habitat my dad started a long time ago without gov involvement, incurring economic loss because of taking land out of production for wildlife. Its the reason deer and turkeys are in the area now. When I was in high school, it was rare to see a deer or tracks and there were no turkeys.
The neighboring hunter is ethical and responsible and is great to have as a neighboring hunter. The way land is being broken up into smaller parcels, there really is no way to expect that one won't be hunting land without close neighboring hunters. If they are legal and hunting their own side of the fence, that is about all they can do. If they are hunting the wrong side of the fence, prosecute them or they will continue to run over their neighbors property.
If a neighbor were harrassing me as some have suggested to do, I guarantee there would be law enforcement involved.