You don't need to worry about this - that sort of barometer drift is totally to be expected. It will reset when you start the motors.
However there is a related problem on the TH where occasionally the altitude will drift by up to 40 or 50 ft, and that is a problem we do need to take notice of, and can solve by starting and stopping the motors a few times. I don't know if that also applies to the Q500, but all the time your drift is just a few feet, this is nothing that need concern you. The barometers we have inside these things are just not that accurate.