Art, I would not remove the fence until the plum is 6-7 feet tall and the the central stem reasonably stout. The deer will browse the secondary branches and the shrub should begin to strongly sucker.
One thing I have noticed from my plums is after 3-4 years there is enough secondary branches that the central leader is left alone.