Already stated here numerous times, but it just really boils down to location. I can see the OP's gripe, being in a legendary area for "trophy" hunting and seeing guys across the fence spend big dollars to come in from out of state just to kill a toddler. They didn't come for the venison, let's be real.
Some people really like antlers, and many don't care about them at all. Takes all kinds I guess. My dad's 1st ever deer hunt was in '89. He got a doe, on his own land. My 1st ever hunt was the following year in '90. I got a yearling buck. It was the 1st rack our family owned. Up 'til that point if my grandpa or dad found a shed while farming it got pitched along with a cuss word into the ravine. Turns out dad and I really took to deer hunting and we quickly liked the beauty and appeal of whitetail deer antlers. In the last 30+ years we've killed a handful of trophies and found quite a few hundreds of nice sheds. My kids are different in that they've always been around the racks and had the sheds to play with. I was worried when my son shot his 1st deer last week (a 2.5 year old buck) that he would compare it to larger deer we own and not appreciate it the way he should. I was totally wrong. His hands are nearly glued to the rack and he said, "I finally have a deer I can call my own."
I'm sure it's debatable if that's a good thing or bad. It's just fueling the fire of liking antlers too much. When you get down to the essence of conservatism vs being silly, lusting after big racks and putting them up on the wall should probably be considered silly. It does waste time and money. The meat is definitely the true prize for us as well. I know plenty of guys who have the work work mentality and they simply go out and get their buck, any buck, and get back to work.
I try to be happy for all my neighbors' kills. I keep the mindset that if they're happy then I'm happy. I just wish a few of them would be happier passing a few up. We live where it's really easy to kill small and medium sized deer but hard to kill big deer. If some of them are hunting the fencelines like they typically do, then I start to feel a little resentment at their decision making.
The last thing I'm about to say I'm not sure how it fits in. Throughout the years others get to decide which animals are trophies and which are meat. Take fishing for example, I really like how bass taste. But I don't get to eat them anymore. People had a say in the matter, and they said bass are trophies and you don't get to keep them. As a kid I got an 18" largemouth which made me proud and we couldn't wait to eat him. Boy I sure caught some grief back at the boat ramps from the fishing club guys. It was legal to keep it back then, but they shamed me. And then they turned most of the lakes into catch and release. Same with muskies. I bought a lot of muskie fishing books as a kid and they have recipes in the back pages on how to cook your muskie. People have since decided muskies are "trophy" fish. If you released the muskie, then your neighbors have a chance also. Kinda sounds like the OP's comment about how we share the neighborhood.
Age keeps coming up. Like trophy status should be measured by age and not antler size. Then you find yourself in a CWD hotbed where the experts tell you old bucks are evil and the worst at spreading and we can't have an old age structure. So now what? I think it was Catscratch who made a comment once about what if we could get bucks to grow large racks by age 2 via stellar nutrition, and could people be happy with that? Could you be happy shooting a 200" 2 yr old? If there was a chance it could help some aspect of the cwd issue. I think that was his point back then, maybe I'm off.
Anyway, just rambling.,.. good debate!