The yeast will be fine - it was never living off your added sugar in the first place.
Flour carries its own natural sugars and starches, and yeast happily feeds on those. That's why lean doughs like bagels and baguettes ferment beautifully with little or no added sugar at all. The essential four remain flour, water, salt, yeast... sugar is a guest, not family.
What added sugar DOES do: speeds fermentation a little, softens the crumb, deepens crust colour, and obviously sweetens. So cutting it has visible effects - trim by about a quarter and the structure won't notice, while the sweetness drops nicely. Go much further and expect a paler crust, slightly slower proof and a leaner chew. Adjust one step at a time and taste.
Cut with confidence. The yeast has its own pantry 🙂
