I've done almost exactly that on several occasions and will continue to in the future. In fact, as I type this, I can see the skulls of 2 different 125-135 class 8 points and 1-148" 8 point I killed. Every one of those deer were killed on years where I passed younger bucks with much bigger racks.
FYI, I'm a little wary of putting foam under a pole barn slab unless you are going to trench in a perimeter footing and insulate that as well. Without the building heat to warm under the slab, frost can creep under the perimeter and start pushing up on the slab. This can wreak havoc on the building.
If you are letting other people shoot whatever they want I see no sense in not letting yourself take a doe for the freezer. One deer will not make or break the situation.