You need to take a holistic look at your land, and not only your land, but also the land surrounding your property. It sounds like you are taking a lot of important things into consideration already. Cover versus food is always a consideration, but I believe that on most properties you can have both through some planning and reasoning.
Finding and identifying an existing shrub or tree is only the beginning. You have to ask yourself if it will survive long term where you found it with or without your help. A good example is some of the hazelnuts on my place. There are some that will eventually be overtaken and shaded out by oaks. I could save the hazelnuts by going in annually and taking out the competition, but then I would be destroying my oaks. A better choice would be to let nature take its course and establish (or promote existing) hazelnuts at another place.
Unless what you are trying to promote is a "climax species" you will always be maintaining it in some way to keep sunlight coming to it. If that is at the edge of a woods or in a fence row, you are a lucky man, because you will be doing the maintenance (providing continued sunlight) by doing other tasks (i.e. mowing a field to keep it a field). This is just one example, but it illustrates how you should be thinking about land succession long term.
Something else you can do is what some refer to as TSI. There can be different reasons for doing TSI, and one of those is giving one climax species an advantage over other climax species by whacking back the ones you don't want. I do this where volunteer sweetgums come up in my oak planting. Both are climax species, and once the oaks get big enough to have a significant advantage, I can walk away. The sweetgums will indeed keep coming back and trying to dominate the forest, but they will never be able to do so, because I gave an incredible advantage to the oaks in the early years.
So, to answer your question, put your effort to what will provide benefit long term. Plant or promote fruit and nut bearing trees where they will survive with minimal effort on your part.
Do you want to be going into a sanctuary area trying to keep something alive, or would you be better off keeping it alive where you aren't disturbing bedding deer?
Also ask yourself about travel patterns of deer, your access points to get to stand location without disturbing them and the all important prevailing wind....
You need a plan, and you need to work the plan. That way you won't have wasted effort....
Like j-bird said, you can get some help here along the way, and maybe even from a fish and wildlife guy in your area. They are very helpful here.
