Try grafting some. Can you go grab a couple trips off your neighbor's tree?
Also, I don't think pawpaw are dioecious, but they tend to be clonal and pollinated by beetles and flies, rather than bees. The pollinators don't move very far, so it's likely your neighbor has two different...