I'm growing in pure beach sand and my 3rd leaf trees probably put out about 2 dozen fruit total this year. This is the first year they fruited and they are just getting started from what I can tell. I expect the output to start increasing rapidly every year.
Sure, good soil definitely helps establish your trees faster, getting them fruiting faster, and they are probably generally healthier producing larger and higher quality fruit. But sometimes you just have to work with what you've got. I do add some quality backfill to my holes at time of planting, I do try to get some fertilizer to the young trees to get them putting on wood, and I do try to give them some lime annually.