To preface I don't know anything about visual cryptography. Also, I'm basing my advice on the research I did while looking for grad schools, and I was primarily interested in US schools with research in arithmetic geometry and algebraic cryptography.
University of Washington has Neil Koblitz who was a big part of the development of elliptic curve cryptography, as well as connections to Microsoft's cryptography research group. University of Philadelphia does a lot of research in relation to arithmetic geometry, and has some grant funded projects on cryptography. The CUNY Graduate Center and the Stevens Institute of Technology have the Algebraic Cryptography Center and weekly seminars specifically on algebraic cryptography. Brown University has Joseph Silverman and has research in cryptography. In Canada University of Waterloo has the Center for Applied Cryptographic Research in collaboration between several of their math and computer science departments. I'm sure there are more, these are just what I remember off the top of my head.