No. Once you have earned voting rewards, they will not decrease. The amount of voting rewards you have earned is expressed as maturity in the NNS app. Rewards = Maturity * Staked ICP. There are conditions that will change your maturity, but they will not change your rewards. Adding more ICP to a neuron and spawning a neuron to collect rewards are both activities that will change your maturity, but neither will change the awards you have earned.
The ICP Guide
The definitive guide to the Internet Computer Protocol (ICP)
