Other than the rainfall, your situation sounds very similar to ours when we started. I'm on a pine farm and there are no row crops within 3 miles. Our deer numbers were very high. The biologist had us shoot every doe we saw. He said that while the deer density in the general area was high, once we established a food plot program with warm season annuals, each time we shot a doe and it left a "hole" in the social structure, when food gets scarce in the summer, deer from the general area would range into our property for food and stay.
After quite a few years of shooting every doe we saw, nature finally helped out. One year, we planted enough that the deer could not keep up and got great plots. We had a mast crop failure that fall, so even with heavy hunting pressure, deer would still visit our plots during daylight hours. We doubled the average number of female deer killed that year. We then had ice storms that winter so the surviving does were not in great shape. While there had been coyotes in the general area for years, they started using our farm regularly that winter. It may have been our fault as we did not burry carcasses. The next year, EHD hit the general area. We had no evidence of it on our farm, but the populations in the general area went down. We had to back off doe harvests the next year.
Over the years we were shooting every doe we saw, we are also doing large scale habitat improvement that increased native foods and increased the BCC.
Buckwheat is a 60 to 90 day crop in terms of food value for deer. We leave it stand until we are ready to do a fall plant and T&M into it. Timing is critical. I'm far enough south in 7a that I'm on the ratty edge of being able to double crop it. I tried for several years. I had to plant the first crop quite early (very late April or early May). Rain was not an issue then, but cold soil was. It would germinate and grow, but the first crop was always lethargic compared to the second. When planting a single crop, late June works well here. We are still getting good rain then, but I always wait for rain in the forecast to plant. The latest we can plant and have buckwheat mature by the fall plant is about the 4th of July here. While buckwheat can germinate with soil temps as low as 45 degrees, the optimal soil temp (not air temp for new folks) is 80 degrees. It really takes off fast in warm soils.
Thanks,
Jack