I wouldn't add any fertilizer, but I'd make sure you've got the big 3 food groups (grass, legume, broadleaf) represented all throughout the growing season. Worms need some legume residue to eat all year. They also need some high carbon residue to live under that will protect from sun, rain, and predators. Calcium is always good for worms, lime is fine too. Just mind your pH so you don't overshoot it and create a chalk layer. I just came across that gem in a book I'm reading now.
I make it a practice to throw on at least 200lbs/ac per year of gypsum while I'm getting the system going. After 3-4 years, you could probably quit with the gypsum because you'll be naturally cycling back all the calcium and sulfate you need out of the parent materials in your soil.