The key is to use the fact that a and b are relatively prime. We show in fact that they must differ by $ 1 (or 0)$. Suppose first that a = b. Then since they are relatively prime they must both be 1. That gives the first answer above. So we may take a > b. Then $ (a - b)^2 < a^2 + b^2 = n$, so if $ a - b$ is not $ 1$, it must have a prime factor which divides ab. But then it must divide a or b and hence both. Contradiction. So $ a = b + 1$. Now $ (b - 1)^2 < b^2 < n$, so any prime factor p of b - 1 must divide $ ab = b(b + 1)$. It cannot divide b (or it would divide b and b - 1 and hence 1), so it must divide b + 1 and hence must be 2. But if 4 divides b - 1, then 4 does not divide b(b - 1), so b - 1 must be 0, 1 or 2. But it is now easy to check the cases a, b = (4, 3), (3, 2), (2, 1).

Chien wrote that (a,b)=(4,3). But then n=25 according to the problem 5|12 which is a contradiction..

I think that $n$ should be a prime number. Let $n$ be composite. If $d$ is a divisor of $n$ then $n/d$ is also a divisor of $n$. Now let $d<\sqrt{n}$. Let $q$ be a prime which divides $d$ . then $q|ab$ and also $q|n$ which follows that $q|a$ and $b$ which is a contradiction. Hence $n$ must be a prime number. Now $p=a^{2}+b^{2}$ and $ab=kq_{1}q_{2}...q_{l}$ where $q_{i}$ is a prime less than or equal to $\sqrt{n}$. now $|a-b|<\sqrt{p}$. But we see that $q_{i}$ does not divide $|a-b|$. This forces that either $|a-b|=1$ or there is only one prime less than $\sqrt{n}$. This prime must be $2$. We know that a prime can be expressed as the sum of two squares iff it is of the form $4k+1$. we see that there is only prime, $5$, satisfying these properties. So $n=5$ is an answer. But if $p=a^{2}+(a+1)^{2}$ then it follows that any prime less than $\sqrt{n}$ must be less than $a+1$ otherwise if q be a prime possessing this then q will not divide $a(a+1)$. Now it follows that $a(a+1)$ is divisible by any prime less than $a+1$ is possible only when $a=2$( one may use Bertrand Conjecture to prove this). If $a=2$ then $n=13$. Hence the answer is $n=5,13$