While I agree with AH-1G in the general sense I will offer this:
When you disable GPS on your Breeze (Or most drones for that matter) you are losing the ability for the drone to "lock" it's position based on satellite signals. Now if the drone is "trimmed" perfectly (or close to it) it will stay pretty much in place while hovering until external forces (wind, etc) act upon it.
And when I talk about trimming, I am referring to all 4 motors spinning at the exact (or **** close) rate so the drone hovers in one place. if any (one or 2) of the motors is spinning at a rate that is faster than the others, the drone will bias or tilt in the direction opposite to it's position on the drone. If it's slower, it will be towards the side the motor is on.
What the GPS mode does is mask any minor (or major) differences in spin rate across all the motors by controlling the individual speed of the motors to a rate that is (nearly) identical. In other words, the drone stays in one place when you have GPS mode enabled and let go of the sticks during a hover.
On some controllers there are trim tabs that allow you to make fine adjustments to the sticks to get to a balanced rate of all the axis's rates which in simple terms means that when you lift you drone off and release the sticks, it will hover in (mostly) one place.
I am not familiar with the Breeze controller so I don't know if you have the ability to adjust this.
If Flying indoors is a thing you REALLY want to do, I would suggest doing it in a VERY big room. IE: Gymnasium, etc. so you can get a feel for which way your Breeze will drift. This will allow you some recovery time to correct for the direction the drone is favoring on liftoff.
Otherwise, follow the short answer provided by AH-1G.... ;-)