No, it will be fine.  You won’t have any problems . They are just eating decomposing vegetation.    I have been growing my own worms,for a few years now. Red wigglers are the best for making castings.  The crawlers go to deep in the soil.    I make them a new bed in the spring and separate them in the fall for the castings.   It is a totally organic soil  filled with nutrients your plants are easily able to absorb.  You cannot over feed.  I don’t plant in pure castings but mix it into my soil.  I Put a couple handfuls in the bottom of my of my hole to. When the roots reach it.   The plant gets another small dose of energy.     So, after a year of research before I even started a worm bed. I learned that with a little top dressing it can also repel spider mites and aphids.  I recycle most of my kitchen scraps and actually have them on a feeding schedule.  In the winter months they are moved to a garage that stays just above freezing.