The flange did NOT come loose from the pipe, it was NEVER connected to it in the first place. The flange was placed over the lead bend and should have been fastened to the floor. Then the lead bend was "flared" over and flattened against the flange. At this stage, I do not see any reason to change anything, other than to screw the flange down securely.
Oh crap. So this entire time, if I built up the floor again, put my plywod, etc down and screwed down the flange, I would be ok? So this entire time worrying about the flange.... Hmmmm some thinking and choices to do.