Unless they are eating into your fruit trees, ants can be a signal that aphids are in your trees. Not a bad thing entirely. They alert you to check for aphid damage, and if it's severe enough to treat for the aphids. A little aphid damage won't hurt a bigger, healthy tree. Younger trees need sprayed so they get a good start in life.