No. Once you have earned voting rewards, they will not decrease. The amount of voting rewards you have earned is expressed as maturity in the NNS app. Rewards = Maturity * Staked ICP. There are conditions that will change your maturity, but they will not change your rewards. Adding more ICP to a neuron and spawning a neuron to collect rewards are both activities that will change your maturity, but neither will change the awards you have earned.