$f(x)=x$ is the only solution. To see this, we begin by noting that $f(p)^2+3pf(p)=x^2$, which means, the discriminant $\Delta=9p^2+4x^2$ of the quadratic, $t^2+3pt-x^2$ must be a perfect square. Studying this, we obtain that, for every $p$ prime, $$ f(p)\in\left\{p,\left(\frac{p-3}{2}\right)^2,\frac{3p^2-6p-3}{2},\left(\frac{3p-1}{2}\right)^2\right\}. $$Now, fix an integer $x$, and let $y=p$ a prime. The object we study is $f(p)^2+2xf(p)+xf(x)$, which has to be a perfect square for every prime $p$. Observe that, for $p$ sufficiently large $$ (f(p)+x-1)^2 <f(p)^2+2xf(p)+xf(x)<(f(p)+x+1)^2. $$Hence, for this fixed value of $x$, we have, by taking $p$ sufficiently large, that $f(p)^2+2xf(p)+xf(x)=(f(p)+x)^2$, giving that $f(x)=x$ for this fixed value of $x$. Repeating this for every $x\in \mathbb{N}$ yields that $f(n)=n,\forall n$ is the only solution.

I hope there are no mistakes. The identity function obviously verifies the given condition and we shall prove that there are no other solutions. Assume that there is some solution $f$ to our problem, for which $f(a)$ is not equal to $a$, for some positive integer $a$. However, the given condition yields : $$ x(f(x)-x) + (x+f(y))^2 $$ is a perfect square for all positive integers $x$ and $y$. By putting $x=a$, we see that if $f$ assumes arbitrarily large values, the previous expression cannot be a perfect square if we let $f(y)$ take a sufficiently large value, as it would lie in between two consecutive squares. Hence, $f$ takes finitely many values. Rewrite the condition as : $$ x(f(x)+2f(y)) + (f(y))^2 $$ is a perfect square for all positive integers $x$ and $y$. However, both $f(x)+2f(y)$ and $(f(y))^2$ assume finitely many (positive integer) values, as $f$ does so as well. This means that the given condition implies the existence of some finite system of linear expressions with positive integer coefficients of the form $ax + b$, such that at least one of them is a perfect square for all positive integers $x$. It is easy to prove that such a system does not exist, and thus our assumption is false, making the identity function the only solution.

Let $P(x,y) : xf(x)+(f(y))^2+2xf(y) $ $P(1,1) \implies f(1)=1 $ Claim : $f(x)=x$ we assume that $f(x)=x (x=1,2,3, ... ,n-1) $ If, $ f(n)<n$ let $ f(n)=n-k $ $P(n,[\frac{1}{4}k]+1) \implies (n+[\frac{1}{4}k]+1)^2-nk$ is perfect square, but it is between two continuous squares $\implies $ contradiction If, $ f(n)>n$ let $ f(n)=n+t $ $P(n,1)$ and $P(n,2) \implies nt+(n+1)^2=a^2$ , $ nt+(n+2)^2=b^2$ $b^2-a^2 = 2n+3$ however since $b>a$, if $a >n+1\implies b^2-a^2 \geq2n+5 $ Therefore, $a=n+1\implies t=0 \implies$ $\boxed{f(x)=x}$

grupyorum wrote: the discriminant $\Delta=9p^2+4x^2$ of the quadratic, $t^2+3pt-x^2$ must be a perfect square. Studying this, we obtain that, for every $p$ prime, $$ f(p)\in\left\{p,\left(\frac{p-3}{2}\right)^2,\frac{3p^2-6p-3}{2},\left(\frac{3p-1}{2}\right)^2\right\}. $$ Why is this true.

Really nice $\color{black}\rule{25cm}{1pt}$ The only answer is $f(x)=x$. Set $x=y$ to get that $f(x)\left( f(x)+3x\right)$ must be a perfect square. Now set $x=p$ to be prime. We have that: $$f(p)\left( f(p) + 3p\right) = k_p^2$$To find out $f(p)$, we use the GCD of $(f(p),3p)$, since we have that this is equivalent to $(f(p),f(p)+3p)$. We have four cases. 1.$(f(p),3p)=1$ We have that $f(p)=x_p^2$ and $f(p)+3p=y_p^2$. Solving this system we have that: \[f(p)=\left( \frac{p-3}{2} \right)^2 \; \; \text{or either} \; \; f(p)=\left(\frac{3p-1}{2} \right)^2\]2.$(f(p),3p)=3$ Let $f(p)=3t_p$. We have that the assertion is turned into $9t_p(t_p+p)$ is a square and this implies that $t_p=x_p^2$ and $t_p+p=y_p^2$. Solving this system we have that: $$f(p)=3\left(\frac{p-1}{2}\right)^2$$3.$(f(p),3p)=p$ Let $f(p)=pt_p$. We have that the assertion is turned into $p^2t_p(t_p+3)$ is a square and this implies that $t_p=x_p^2$ and $t_p+3=y_p^2$ Solving the system we have that: $$f(p)=p$$4.$(f(p),3p)=3p$ Let $f(p)=3pt_p$. The assertion gets turned into $9p^2t_p(t_p+1)$ is a square and this implies that $t_p=x_p^2$ and $t_p+1=y_p^2$. But this system doesn't have a solution. Now let's fix some $x$. No matter how and which form $f(p)$ has, for a large enough $p$ we must have that: $$(f(p)+x-1)^2<xf(x)+f(p)^2+2xf(p) < (f(p)+x+1)^2$$thus we have that $xf(x)+f(p)^2+2xf(p)=(f(p)+x)^2$ and this implies that $xf(x)=x^2$, which gives us $f(x)=x$.

The only solution is $f(x) = x$, which indeed works. From $P(x, x)$, we have $f(x)^2 + 3xf(x)$ is a perfect square for all $x$. Since $(f(1) + 1)^2 \leq f(1)^2 + 3f(1) < (f(1) + 2)^2$, then $(f(1) + 1)^2 = f(1)^2 + 3f(1)$, so $f(1) = 1$. Similarly, $(f(2) + 1)^2 < f(2)^2 + 6f(2) < (f(2) + 3)^2$, so $f(2) = 2$. From $P(x, 1)$ and $P(x, 2)$, both $xf(x) + 2x + 1$ and $xf(x) + 4x + 4$ are perfect squares. Assume $f(x) > x$. Let $xf(x) + 2x + 1 = k^2$ (where $k > 0$), then $k > x + 1$, however $$k^2 < xf(x) + 4x + 4 = k^2 + 2x + 3 < k^2 + 2k + 1 = (k + 1)^2,$$contradiction. Hence, $f(x) \leq x$ for all $x$. Now, let $p$ be a prime. We will show that $f(p) = p$. From $P(p, 1)$, $pf(p) + 2p + 1 = k^2$ for some positive integer $k$, then $p(f(p) + 2) = (k - 1)(k + 1)$, so either $p \mid k - 1$ or $p \mid k + 1$. Since $k^2 = pf(p) + 2p + 1 \leq (p + 1)^2$, we must have either $k = p - 1$ or $k = p + 1$. If $k = p - 1$, then $p(f(p) + 2) = p(p - 2)$, so $f(p) = p - 4$. However, $p^2 < pf(p) + 4p + 4 = p^2 + 4 < (p + 1)^2$, so $P(p, 2)$ gives a contradiction. Hence, $f(p) = p$, as desired. Therefore, from $P(p, x)$ (where $p$ is prime and $x$ is fixed) we have $p^2 + 2pf(x) + xf(x) = (p + f(x))^2 + x(f(x) - x)$ is a perfect square. By taking $p$ sufficiently large, we will have $x(x - f(x)) < 2(p + f(x)) - 1$, so $$(p + f(x))^2 \geq p^2 + 2pf(x) + xf(x) > (p + f(x))^2 - 2(p + f(x)) + 1 = (p + f(x) - 1)^2.$$Hence, $(p + f(x))^2 = p^2 + 2pf(x) + xf(x)$, therefore $f(x) = x$ for all $x$.

silouan wrote: Find all functions $f:\mathbb{Z}_{>0}\mapsto\mathbb{Z}_{>0}$ such that $$xf(x)+(f(y))^2+2xf(y)$$is perfect square for all positive integers $x,y$. **This problem was proposed by me for the BMO 2017 and it was shortlisted. We then used it in our TST. My proof seems to be different . . $Claim_1$, we prove $f(x)$ is not bounded. assume for the sake of contradiction that $f(x) \in \{a_1<...<a_n\}$ for some natural $n$ . rewrite the statement as $$g(m,n)^2=nf(n)+2nf(m)+f(m)^2$$now take $m=n$ so we have $$T(m)^2=g(m,m)^2=f(m)(3m+f(m))$$now take $n$ distinct prime numbers $p_1,p_2,...,p_n$ which all of them are bigger than $a_n^2+1$. and by chinese reminder theorem construct $m$ such that $$V_{p_i}(3m+a_i)=1$$this clearly gives us contradiction . and hence the function is not bounded. . now we shall rewrite $g(m,n)$ as : $$g(m,n)=(f(m)+n)^2+(nf(n)-n^2)$$take $m$ such that $f(m)> 2nf(n)+2n^2$. this forces to have $nf(n)=n^2$ and hence $$f(n)=n$$for all positive integers $n$ , . . note: with this method the problem can also be solved from integers to themselves.

Interesting FE $P(1,1) : f(1)^2 + 3f(1) = k^2$ and we have $f(1)^2 < k^2 < (f(1)+2)^2 \implies k = f(1)+1 \implies f(1) = 1$ $P(n,1) : nf(n) + 2n + 1$ is square Claim $: f(x) \le x$. Proof $:$ Assume not so there exists $t$ such that $f(t) = t + u$. $P(t,1) , P(t,2) : (t+1)^2 + tu = x^2 , (t+2)^2 + tu = y^2 \implies y^2 - x^2 = 2n+3$ but Also we have $x > t+1$ and $y > t+2$ which gives contradiction since difference of squares is increasing. Claim $: f(p) = p$ for any prime $p$. Proof $:$ Note that $P(p,1) : pf(p) + 2p + 1 = r^2 \implies p(f(p) + 2) = (r-1)(r+1) \implies p | (r-1)$ or $(r+1)$ and since $pf(p) + 2p + 1 = r^2 \le (p+1)^2$ we have $p = r-1$ or $r+1$. if $p = r+1$ then $p(f(p)+2) = (p-2)p \implies f(p) = p-4$ so $P(p,2) : pf(p) + 4p + 4 = p^2 + 4$ which gives contradiction so $p = r-1$ which implies that $f(p) = p$. Note that $p^2 + 2px + x^2 + x(x-f(x)) \ge p^2 + 2px + xf(x) + x(x-f(x)) = (p+x)^2$ and Note that for large enough $p$ we have $(p+x+1)^2 > p^2 + 2px + x^2 + x(x-f(x))$ so $p^2 + 2px + x^2 + x(x-f(x)) = p^2 + 2px + xf(x) + x(x-f(x)) \implies f(x) = x$