That is odd behavior. The IRS can sense low level altitude in daylight or nighttime conditions. Are you flying with GPS enabled or disabled? If GPS is disabled, the Flow Sensor will be enabled and needs enough light to detect movement of the aircraft.
You do not need the controller to fly at night. You do not need a Part 107 certification to fly at night. If you had a 107 cert you would have to apply for a waiver to be able to fly at night.
My suggestion is to be sure GPS is enabled and give the Breeze enough time on the ground to get a good GPS fix. Small lights on the legs will help you with aircraft orientation. Possibly white in front and red in back or green on the right and red on the left.
Your control signals should not differ between daylight and darkness. In fact the phone is easier to see at night than in bright sunlight for seeing where the soft-sticks are.
Of course with the new FAA Reauthorization Act of 2018, things will be changing soon and new rules will be coming out in the next few months assuming you are flying in the US.