I'm a beef producer first and hunter second. Have never been anal about species in my foodplots... but I've been including annual ryegrass - along with wheat, occasionally rye, rarely oats, and various clovers, in foodplots for over 45 years. Annual ryegrass may be an issue for some of y'all, but not for me. But... I'll admit up front, that in most cases, after the end of hunting season and before things headed out, we usually opened the gates and allowed the cows to graze foodplots - sometimes multiple times. Did I get some 'volunteer' ryegrass the next fall, when I replanted? Yes; but I welcomed it. YMMV.
Some confusion invariably arises in these threads, as some folks don't know the difference between (winter/cereal) rye and annual ryegrass... or, other knowledgeable folks say 'annual rye', when they mean 'annual ryegrass', adding to the confusion potential. The two are not the same...
Vomitoxin can form in grain crops, like wheat and (cereal) rye... but to my knowledge, it is not an issue with annual ryegrass. Have never noticed deer feeding on the mature grain heads of wheat or rye, and vomitoxin is not present in the foliage... only in the grains. I suppose it could be an issue for turkeys or other birds feeding on the seed after maturation.
I have no experience feeding vomitoxin-contaminated grain to poultry, but in cattle & hogs, the only significant issue is feed refusal... and thus, lowered rate of gain.