you should be adjusting the polarizer whilst looking through the viewfinder, so you don't really have to "go back to the frame". You are constantly looking at your subject whilst adjusting the CPL. If you don't move back or forth during this time, your focusing shouldn't be out, or should only be SLIGHTLY out. Be careful not to rotate the focusing ring whilst rotating the CPL, that's all...