I don't let new fruit trees produce fruit for the first three years or so, I want the trees energy to produce wood/roots and leaves when they are young.
I don't get to hung up on the blossoms but will pinch the fruit off if it starts. First few years I am more interested in getting the tree growing strong and shaped correct.