We're pretty blessed, but lose a lot of 3 year olds to the neighbors. Last year I think the neighbors shot 4-5 that were 3 and younger. They were deer we either had on cam or had passed in person. It happens. All hunters have different goals in the woods, and if you're doing it legally, I can't stop you.
That said, our biggest frustration has been all buck hunting and no one shooting does. We generally have a couple 4 or 5 year olds to chase, but we cannot get them to the size we think they should be.
a 2-4 year olds we shot went right at pope and young, while the 2 5 year olds were 132" and 140" respectfully. Those are the oldest deer in the woods and they didn't grow much from last year. Age wise they should be 150+ easily, but we can't seem to get them there. But we're working on that. the last 4 years or so we've done substantial habitat work and hoping to turn a corner. But it's tough when the neighbors shoot teh 135" 3 year old that has the potential to blow up at 4 or 5. To most hunters a 3 year old looks good enough to kill, but is also dumb enough to be visible.
If you wanna shoot the big mature deer, you have to be willing to only kill one every 2-4 years in my area. I'm ok with that, just sucks when that mature deer only scores 135". However, I've been teaching myself to hunt for age, not antler size