I don't think it'll be a problem. While it's theoretically possible for the seals to move up and down as the forks work, it's so unlikely that you may as well consider it impossible. There's no way the friction between seal lip and fork stanchion will overcome the friction between the back of the seal against the inside of the fork slider.
But if you think the gap between seal and circlip will give you sleepless nights, just make a spacer to fill that gap. Take to your old seals with a grinder to thin them down to 4mm.