Well the holes chewed through the leaves are from bug damage, either worms like tent worms or beetles like, chafers or Japanese beetles.
But, that did not cause the leaves to turn brown. Leaves dying means stress (drought) or damage to roots or damage to the bark below (girtling).
You can SAY you're not going to baby them, but if you don't while they are young you might as well not plant them at all, or plant 10 times as many as you want to live.