I'll give you a few examples of how I've seen this stuff handled. I think it really depends on the situation. When my dad bought our stuff, there was a guy who hunted it and basically said "well I've been doing it for 10 years, so you don't have the right to tell me no." Obviously that kind of attitude wasn't going to fly, and after a few years he got the picture.
I am certainly someone who benefits from using land that I don't own (my dad owns our land, but I mean outside of the family). My neighbor has some 300 acres, and he hunts mainly on the complete oposite side. I also am right next to state land, and near one of two access points. He doesn't let me go INTO it and hunt, but we did set up a blind (with his permission and overview) that I can use. It is only 50-60 yards from his line. In between our properties is 100 feet that someone hunts. This person is a real pain for both of us. In this case, I act as eyes on this area for BOTH of our properties. He doesn't hunt it because he has MUCH better spots with MUCH less pressure. If he were, however, to want to go there, he would absolutely be friendly and say I'm going to sit it. It's still HIS land, and I understand that as his neighbor. This guy is also someone who has helped me with EVERYTHING in the way of hunting. He helped hang my first sets when I began bowhunting at 13. He helped me food plot using his tractor. My dad isn't really a hunter, and while he enjoys going out and sitting, the habitat stuff isn't his specialty. Both my neighbor and I benefit from this. When my neighbor goes to Florida (Jan-April) he lets me rabbit hunt. I look around for sheds and carcasses and let him know if I see anything out of place. He loves that I can enjoy it, and that I can keep an eye on it. Similarly, he lets my turkey hunt his whole property, except for his sanctuaries, in the spring. Same thing, he likes me getting out there and looking around. He's 70+, and although he still climbs in the bucket of his skidsteer to cut limbs, he's not as mobile as he once was.
I have a neighbor that has about one acre near the road. I have a stand about a hundred yards in off of the road. I have asked permission to walk his lot (basically down his driveway) to get to my stand that is 20-30 yards onto mine. He doesn't hunt it (although I think I did see him out there this winter, and he was perhaps on ours...). He said yes, and thanked me for asking. Now, I'm not hunting his land so this is different, but my only other way to get there is to make a hell of a trail through a nice thick area that regularly holds deer movement, OR walk right through the middle of my food plot. This is an evening stand, so many times I walk through the plot to get TO the stand, and then go out the back onto his.
We have another few acres in a swamp directly across the road from our main property. We then have an ajoining 100 feet, with another 300 foot section that is owned by someone else. They have owned this piece for decades, and don't use it at all. We offered to buy it from them, and they said no. We asked to hunt it, and they said ABSOLUTELY! Again, I'm benefiting from someone else allowing my use.
Now, these were all of ME and neighbors... so here is one for my BIL.
My brother in law's family grew up hunting a 40. They had pretty good neighbor's except for one that didn't let them go retrieve the first deer my BIL shot when he was 14. The deer was in view. Basically the guy was just a dick. My BIL's father offered the land up for spring gobbler season to youth. I'm not sure if he went with them or not. Now, he also lets his family hunt there. I think total there is 5-6 guys on there opening day of rifle. Needless to say, usually no more than 2 get a deer. His main goal of the land was not harvesting trophy deer, but rather to have it for his entire family to enjoy and spend time together. In this situation, it makes a bit more sense to allow people to use it, AT YOUR DISCRETION.
Now, as for what I would do in your situation. For me, I think there are a few things I would take into consideration. I am assuming that since you are open to entertaining the idea of letting anyone use it you want to be a friendly and generous neighbor. Not that you can't be that without letting them use it, but that is just the impression I'm getting. I might be completely wrong, and there is nothing wrong with the opposite. Although I don't really do it myself, this is what I would think about.
1. If I am to let anyone use it, it must be crystal clear that they play by my rules. As Nofo said, good fences make good neighbors, and I think one type of "fence" are clear and concise ground rules.
2.What are my goals for the property? Am I trying to grow insane racks and ONLY use this for hunting and not general recreation? Will I myself be doing other hunting?
3. What are they asking, how are they asking?
The big thing is are they being respectful of this. I know that as an owner you are under no obligation to help them. That being said, as someone who has benefited form others allowing my use, I know that being respectful, honest, and completely up front with what you are wanting to do goes a long way to their ability to fit you into their plans. If someone is going to be standoffish, then screw them.
4. Will what they are asking to do have an impact on you. I think you're already onto this one. Like in the case of the ATVs. Or the grouse hunting. I look at my neighbor allowing me to use his land to hunt. What I do there has little to nothing to do with the deer he sees in his other blind. He has more food, more cover, and a better setup. Now, if I get pictures of one of his big boys, I show them to him, and many times he says OK, now let's stay the hell out of there. In that case it might affect him, so he applies a different approach.
I think overall you get it. If you decide to let anyone in I think It then depends on the situation, who is asking, what they are asking, how they are asking, and how it will affect you. I the biggest thing is having non-negotiable ground rules that are dictated by you and will be enforced. You shouldn't have to, but make it easy for people to know what they are not to do. The rest you can feel out on a case by case basis. Don't hesitate to ask for some help with something. Someone needs to build a relationship with you in order for you to trust them, this is no different.
I know that I have different thoughts on this than some, but I always find that I enjoy my things more when I can see others finding joy in them as well. If I've got something I like, I'm usually more than happy to share it with someone, so long as it doesn't interfere with my own expectations.
So at the end of this long winded post I'd say that my advice would be to say nobody using it for hunting for at least the first year. Maybe special cases can apply for youth hunters or if you supervise them. You can use the "I want to get a sense of what is really here" line, and if they have a problem with that, then they can expect themselves to be removed from any and all consideration. The only WRONG answer is one that ambiguous and allows wiggle room for others to determine your rules.
