"Yeast grows from sugar."

This is a common misconception. Yes, yeast do certainly like to eat sugar, but they can eat other things when necessary and they do. You can't eliminate them by diet alone.