Yes I do concern myself with residual 2-4,D. That is why I don't like to use it unless I have to. Of course, it also depends upon what you are planting afterwards but for following with brassicas (or beans or sugar beets, etc) I will generally spray (1 pint/acre 2-4,D and Gly) and then wait 10-14 days before planting. Better to be safe than sorry. If you are going to spray at 1 quart per acre, of course, you need to wait longer.
This is precisely why I am hoping that I can still get the N boost from my cover crops with Crimson, Balsana and Hairy Vetch and maybe not have to use 2-4,D to terminate them.