$(x,y,p)=(2,1,5),(1,2,5)$ are the only solutions, where I take $p=x^2+y^2$.
Note that $p=2$ is impossible, hence $p>3$. Thus exactly one of $x,y$ is even, assume it is $x$. Now if $d=(x,y)$ then using $d\mid p$ and $d<p$ it follows $d=1$, namely $x$ and $y$ are coprime. Hence, $x=2a^2$ and $y=b^2$ for some $a,b\in\mathbb{N}$. With this we obtain
\[
p=4a^4+b^4 = \left(2a^2-2ab+b^2\right)\left(2a^2+2ab+b^2\right).
\]Since $p$ is a prime, this is possible only when
\[
2a^2-2ab+b^2=a^2+(a-b)^2 =1 \implies a=b=1,
\]where the latter holds as $a\ge 1$. Hence $a=b=1$ and $p=5$ is the only solution, for which $(x,y)=(2,1)$.