I really want to pursue a PhD in mathematics. Some background info, I have my B.S. in mathematics with minors in chemistry, biology, and computer science. I graduated with a 3.8 GPA and had a very good scholarship for my undergrad. This semester I am graduating with my M.S. in computer science. I have two papers under review with journals in fields not related to mathematics (virology and CS). My GPA is good, but sometimes I truly feel like I didn't truly deserve it, maybe will make sense below

Now my worries. Everyone tells me PhD programs for math are impossible to get into. I am really worried. I also wonder if I am competent to be successful in a PhD program. The reason why I wonder is because initially I struggled a lot with pure mathematics. Abstract algebra was hell for me (I got a B in the class), and then real analysis was so difficult for me (Though I got an A but I truly believe it's because the prof took some mercy/pity on me because I just worked so damn hard in the class). I really gave up on math after those courses. But then suddenly, I took topology this semester (this is my last semester of M.S. and I begged them to let me take more math courses) and it all "clicked". Real analysis, algebra, topology, these classes all became very easy for me suddenly. Topology is like cake for me, and I can't for the life of me understand why the other classes were so difficult for me before. The reason I think it is easy for me now is just because I put such a huge amount of time and effort into these classes. I can't stress enough how none of it came naturally to me, but only after a lot of dedication and hard work. But now honestly in topology, I breeze through the course although I find it completely fascinating so I just study so much and love every moment of it. I just don't find it difficult at all (note, I don't say any of these thing to brag, just trying to give a complete picture of my current situation. Again, I really sucked so much at all of this originally which is why I'm confused)

Anyway, is it even worth it to pursue a PhD in mathematics or do you need to be one of these people who just don't have to work and it's all cake from the first moment?

Also, I have had an "interesting" academic journey. I come from a low income area and dropped out of high school and left home very young. When I began college, I started in intermediate algebra. So I have come a very long way and because of this sometimes I feel I am very behind because I have had to teach myself a lot. My peers in college had a huge head up on me, when I started I had literally never taken a single course in chemistry/physics/calculus/biology, anything beyond basic elementary algebra. I have always had a difficult time learning in lectures, I don't know why, and so during my B.S. and M.S. I pretty much never went to lectures and opted to teach myself from text books. The classes that were exceptions were abstract algebra and real analysis, I really depended on bugging my profs day and night with questions over that material. Anyway, I don't know if this is at all relevant just that, I really feel I am behind even still today and just... I don't have much confidence in mathematics even though I really love it more than I can say. Pure mathematics gives me such a thrill and I am so curious in many areas.

I'll also add that the reason I went for my M.S. in CS rather than math was because I just felt like I would never be able to make it in math after struggling with real analysis so much. I really learned how much I don't enjoy CS and just have no desire to stay in this field. But I feel like this M.S. is going to make me look bad for abandoning math I guess

EDIT: The math dept. here is small and not many classes are offered. The only pure math courses I've taken: some intro to logic course, abstract algebra, real analysis, functional analysis, topology. I am self studying complex analysis because I really want to learn it but it's never offered here. I'd be happy to self study any other subjects that are fascinating like this but I don't know what to go after next! Also, I really *really* hate statistics. Not sure if that is relevant.