I guess I am someone who doesnt like to involve police, lawyers, or court. If it were me, I would ask him to meet you out there, and explain your concerns on the dam, and how it can be righted out of court. I have learned over the years that no one will win if it has to go to court.
It is possible that the old timer wasnt there, and wasnt aware the construction crew took the dirt from you, and they took it apon themselves to find the easiest dirt. In the grand scheme of things, how will this effect your land negatively, then ask him to correct that. You cant undo what was done, but you can fix what was done wrong.
If that means him paying for some new trees for cover, and hauling in a few loads of dirt to redirect the water back to where it is suppose to be, then that is what is fair. No amount of money will make what happen, not happen. Fix the issue, discuss with him to pay for it, and move on with life.
It may not be a bad idea just to ask a DNR agent to join you two when you meet to look at it, to discuss any future liability, and if it was done correctly in the DNR's eyes.