I have this exact problem and obviously many guys interested in growing older age class bucks do. One thing that bothers me is how over the years the state has become so much more lenient on season lengths and weapons. In MN, we used to choose between a 2 day gun, a 4 day gun, or a MZ season. You could pick 1 not all. As time went on, they made it a 9 day gun season or 16, AND you can also hunt the 16 day MZ season. If youre willing to pay, you can about hunt whenever you want with whatever you want. Many bucks were smart enough to lay low for a 4 day gun season... but 16... not gonna happen.
Most hunters dont care, and they arent interested in making the effort to learn. For example... A deer with a "grey" face must me 8 years old etc etc. I watch the deer very closely in about a 12 square mile area around my home. On most years I will know of 2 bucks that make 4 yrs old in that area. In the last 14 years I can only think of about 5 deer that have made it to 5 years old, and I believe all of them were killed at age 5. I dont know of a buck thats lived to 6. Also, at least 3 of the 5, 5 yr olds, were killed by cars and not hunters!
In the 1000 acres section I hunt, a buck makes it to 4 1/2 about every 4 years. So for me, it makes it where now I rarely hunt 3 out of 4 years. I work harder to keep the deer alive than killing them on those years.
Its hard and its frustrating. You have to have hope that some can make it and your efforts are worth it or it is all for nothing. I cant feel good about shooting a 3 yr old anymore, I just cant. I dont like it but its the way it is. Getting pictures, finding sheds, creating habitat and food, those things now fill the void of NOT killing a deer so I find new satisfaction in them. I find it pure hypocrisy to preach the importance of passing 2 and 3 yr old deer only to kill one myself, that would show a serious lack of integrity. I have no problem with hunters shooting whatever they legally want. BUT dont complain about everyone killing little bucks and then killing one your self! At that moment you become part of the problem.
Have faith in what you are creating there and feel good about it. If you can't do that, then get the F#$% OUT and find a new spot! Life is too short!