If there are a lot of trees or buildings close by, you may be getting reflected GPS signals that can cause a slow drift. If the yard is enclosed by a metal fence that can also affect a slow drift.
I was recently flying at a Veteran’s Memorial near my home. The Breeze kept drifting and had me puzzled as to why, until I realized it was the large amount of rebar in the concrete around the center of the monument.
On a calm day take the Breeze out in the yard and fly it with the GPS turned off. Keep it under 10 feet so the IR and Flow sensors keep it stable. See if the drifting is better or worse than with GPS turned on.