Topics which synthesize analytic and algebraic techniques are typically the most difficult, because most people have trouble with one or the other. Some such areas, all of which also have an enormous number of prerequisites, are algebraic geometry, algebraic number theory, ergodic theory and arithmetic combinatorics. However, analytic number theory is not a comparatively difficult field; it is simply that the problems which still exist in number theory that are anticipated to be solvable analytically are rather ridiculous. Neither is proper algebraic topology, although an increasing number of problems in algebraic topology have been solved by considering them in the context of geometric topology, which is most definitely a very difficult field. But in nearly any field, there are sub-fields which can be enormously difficult to work in. Some examples in fields that were not mentioned above might be model theory (in logic), Lawvere theory (in category theory), mechanism design theory and latin squares (in combinatorics), disk algebra (in complex analysis), and L-Theory (in K-Theory). In noncommutative algebra and algebraic number theory, there are a number of structures and techniques which are very specialized and extremely difficult: nonassociative rings, quasigroups, cyclotomic fields, Iwasawa Theory, etc.
If we are talking about at least moderately broad fields, I would say that algebraic geometry, algebraic number theory, ergodic theory and arithmetic combinatorics are the most difficult fields to work in. Algebraic number theory is a bit of an odd man out, though; the material is certainly difficult, but the difficulty with algebraic number theory really lies in the fact that you need to be a true master in a huge number of areas, any one of which is a field in and of itself. For instance, a great algebraic number theorist will be an expert in algebraic geometry, but the converse is not necessarily true. If we are allowing sub-fields, then I would vote for geometric class field theory, crystalline cohomology, Iwasawa theory, geometry of numbers, descent theory, capacity theory on algebraic curves, arakelov geometry, latin squares and Milnor K-Theory. I know very little about mathematical physics, so I can't comment on anything from that field. Some problems in combinatorial game theory and geometric group theory are also intractable, but I would not say that those are exceptionally difficult fields (merely that exceptionally difficult problems can arise in them).
A rule of thumb: if you're wanting to avoid difficult subjects, then avoid anything with "cohomology," "arithmetic" or "geometric" in its title.
A final note: the Inverse Galois Problem and the Erdos Conjecture on Arithmetic Progressions are two of the most difficult problems in mathematics, and neither are strictly from any of the topics listed above. Every field has extremely difficult problems in it. Somebody who is an expert in algebraic geometry, for instance, would likely have less success on a big problem in the field than somebody who is moderately proficient in algebraic geometry, but is also proficient in a number of related fields. Big problems in common topics are not going to be solved by using techniques specific to that area; if they could be, then they already would have been. It is more useful to know some about many fields, so you can see a problem in a potentially new way than people who have already looked at it with a more narrow lens.