Is there a point in time when physicians should just pay off their remaining loans if they have the means to do so?

Being rid of student loans once and for all can be a gratifying feeling, but will it jeopardize your retirement?