So it seems like I will be attending a 2 year master's program in pure mathematics. I know, I should focus on math and focus on getting good grades and doing research so that I can get into somewhere after 2 years.

Well, I am very much planning on doing that. But at the same time, I feel like I should know some physics to do the kind of research that I want to do, and I feel like it would help me out. I am primarily interested in algebraic geometry and topology. For PhD, I really want to do mathematical physics or at least applied math with focus on physical applications.

I stopped taking physics courses as an undergrad after the whole intro sequence (got an A in all). Right now, my plan is to take 1 physics course each semester (so 4 physics courses total: Classical Mechanics, E&M, General Relativity, Quantum Mechanics). At the very least, I want to take General Relativity and Quantum Mechanics. Do you think this is a foolish move if I want to get into a good mathematics PhD programs after my masters? Do you think it would be a distraction and time consuming?