1. Despite what some professors say, prestige matters a lot.
Top schools tend to attract top talents. Being mentored by top talents and having access to contacts and resources matters a lot for your career. My personal experience from doing an undergrad at tier 2 then masters at tier 1 is that there are simply a LOT more opportunities and driven people at tier 1.
Also, the sad fact is that an ivy league PhD has an advantage over a tier 2 PhD when it comes to job applications in or outside of academia. There are always exceptions. E.g. those who did well despite doing a PhD at a tier 3. But it is not prudent to plan to be the exception.
2. After academic achievements, fit is paramount.
Everyone applying will have stellar academic achievements. Then next key thing is "fit". Does the department do the kind of research you want to do? Have you been doing the same kind of research? Do you have clear research goals? Do you have any achievements in those research? Have you shown interest and expertise in that area?
The more you can convince them you are a good fit the better. I know people with amazing profile who get rejected because their research interest do not fit. I was rejected by an ivy league because I did not fit (research direction was rather undeveloped a year ago).
3. Your publications give you a good advantage.
Your supervisor is right: having publication is a very good advantage. Go for it!!