The clear answer is "it depends". I have Allegheny Chinquapins growing natively on my farm and there is very little browsing pressure on them. They are in the chestnut family. I planted Dunstan chestnuts and without tubes the deer browsed them into bushes. I have also collected nuts from my ACs and started new trees. I planted them with not protection and none were browsed. There is one difference which is location. The chestnuts were planted along the edges of clover fields and the ACs were planted (along with other dunstans) along a logging road with no clover. Neither the dunstans nor the ACs along that road were browsed.
My theory, for what it is worth, is this. Deer are browsers by nature. They come to clover fields to feed and it is just their nature to take a bite of any young tree they come across as the feed. The deer are not coming to browse the trees as they have plenty of browse available in the bedding areas. They were coming for the clover and the chestnuts were there.
So, it depends on your deer densities, planting location, and foods available. If it were me, I'd protect most and leave a few unprotected to see how deer relate to them fro future reference.
Thanks,
Jack