Okay I have good news!!
I was able to finally get a good compass calibration, took about 13 tries, but I got the green light on the H letting me know it was successful. I also calibrated the accelometer as well.
I flew it strictly in angle mode and everything worked great, no more drifting...