Case colors fade over time for a variety of reasons. In my opinion long term exposure to light is a far bigger threat than occasionally shooting it.

If you truly "hunt" with it, they will wear. The traditional carry is across the arm. Also carrying in the hand around the receiver, or over the soldier. I hunt with mine carrying them eight hours at a time and they do wear. A simple coat of clear lacquer that one can buy at Lowes will protect them. If you just fondle it, they will last forever. That's why guns used have case colors only in "protected areas." You'll see that phrase often, and a rating of 10% to 20% case colors. Lacquer comes off easily with acetone. Up to you.