I'm an ex-actuary who was in a similar position. I graduated in 2012 with a math degree, earned my ASA, and applied last fall. I was accepted to several good programs (you dig up my profile here
for more info) so it's definitely possible to get into a good program after an extended stay in industry. However, my background is in pure math and I was a fairly talented student, so maybe I'm not the exactly the example you're looking for.
I think you'll have to be strategic about your application. I've heard the mGRE carries a non-negligible, but not considerable, amount of weight in the eyes of admission committees. For you it might carry more since it's a better indication of your current mathematical abilities, but this is just speculation on my part. I think you'll find the actuarial exams will give you a leg up in terms of study discipline and solving tricky problems under time pressure (that was my experience at least).
I would also be hesitant to get letters of recommendation from supervisors and/or actuaries. The letters are usually one of the most important, if not the most important, part of you application and they should speak to you ability to complete a math graduate program. Most people in industry cannot do this. Getting letters from your profs of 10+ years ago will also be tricky since they might not remember you well enough to speak to your abilities. Another concern is that they might not be able to speak to your performance in areas of math admission committees care about. It would probably be difficult for a professor of actuarial science to say much about one's abilities in subjects relevant to a pure math.
For what it's worth, I think going for a masters or taking courses at a local university might be a good idea. This will allow you to shore up any deficiencies in your background, form relationships with professors who can write letters, and give admission committees more recent data with which to make decisions. Some of what I've said also depends on you mathematical interests. If you're looking to get into applied math then maybe your work experience will be viewed favorably and your actuarial knowledge will be relevant.
I hope that was helpful. Feel free to PM me if about this if you'd like.