You can just notice that the maximum is achieved when x+3y=10. Then just substitute x=10-3y in x^2+y^2 and maximize it in the interval y \in [0, 10/3]. Take a square root of the result and that's it.
First sketch the domain. It is a triangle in quarant I. Then you substitude the vertices into the given function. No need to try (0, 0) 'cause the function is bigger when x and y are getting bigger.
The function is simply the distance from the origin. Our domain is a triangle in the first quadrant, with the furthest vertex at (10,0). Thus, the answer is 10.