Here is the easy part of the solution: (1) Set $x=y=1$ and $a:=f(1)$. Then $a^2+3a$ is some square $b^2$. The value $b=a$ is too small (as $a^2+3a>a^2$) and the value $b=a+2$ is too large (as $a^2+3a<(a+2)^2$). Hence $b=a+1$, and $a^2+3a=(a+1)^2$ with $a=1$. Hence $f(1)=1$. (2) Set $x=y=2$ and $a:=f(2)$. Then $a^2+5a$ is some square $b^2$. The value $b=a$ is too small (as $a^2+6a>a^2$) and the value $b=a+3$ is too large (as $a^2+6a<(a+3)^2$). The value $b=a+1$ does not work, as $a^2+6a=(a+1)^2$ has no integer solution. Hence $b=a+2$, and $a^2+6a=(a+2)^2$ with $a=2$. Hence $f(2)=2$. (3) By setting $y=1$, we get that $xf(x)+2x+1$ is a square $a^2$ By setting $y=2$, we get that $xf(x)+4x+4$ is a square $b^2\ge(a+1)^2$. Then $(xf(x)+4x+4)-(xf(x)+2x+1)\ge (a+1)^2-a^2$, and hence $2x+3\ge 2a+1$ and $a\le x+1$. Finally $xf(x)+2x+1\le(x+1)^2$ yields $f(x)\le x$ for all positive integers $x$.

Notice f(1)^2<f(1)(f(1)+3)<(f(1)+2)^2 so f(1)=1 and (f(2)+1)^2<f(2)(f(2)+6)<(f(2)+3)^2 so f(2)=3. Notice P(x,1), P(x,2) we obtain f(x)<=x. Claim: f is unbound. Assume f bounded above by c, choose f(y)=c. Notice f is integer sequence, bounded above hence it must exist infinitely f(x+1)>=f(x) or else it exist M then f strictly decreasing to below 0. Notice xf(x)+c^2+2xc=a^2, (x+1)f(x+1)+c^2+2(x+1)c=b^2 and b^2>=(a+1)^2 (f(x+1)>=f(x)) thus f(x+1)-f(x)+2c>=2a+1 so 3c >= 2a but choose x large enough (infinitely), a->oo so absurd. Now we can fix x, pick f(y) = t large enough that (t+x-1)^2 < (t + x)^2 + x(f(x) - x) < (t+x+1)^2 satisfy so f(x) = x, done!

nguyenhaan2209 wrote: Notice f(1)^2<f(1)(f(1)+3)<(f(1)+2)^2 so f(1)=1 and (f(2)+1)^2<f(2)(f(2)+6)<(f(2)+3)^2 so f(2)=3. Notice P(x,1), P(x,2) we obtain f(x)<=x. Claim: f is unbound. Assume f bounded above by c, choose f(y)=c. Notice xf(x)+c^2+2xc=a^2, (x+1)f(x+1)+c^2+2(x+1)c=b^2 and b^2>=(a+1)^2 thus f(x+1)-f(x)+2c>=2a+1 so 3c>=2a but choose x large enough, absurd. Now we can fix x, pick f(y) = t large enough that (t+x-1)^2 < (t + x)^2 + x(f(x) - x) < (t+x+1)^2 satisfy so f(x) = x, done! Notice xf(x)+c^2+2xc=a^2, (x+1)f(x+1)+c^2+2(x+1)c=b^2 and b^2>=(a+1)^2 The inequality $b^2>=(a+1)^2$ is not justified, as you do not know how $f(x)$ and $f(x+1)$ relate

https://artofproblemsolving.com/community/c6h1814557p12100942

First we prove that $f$ is unbounded$.$ Assume to the contrary that it is bounded by a constant $M$ and let $x=p$ be a large prime$.$ Now rewrite the condition as $$pf(p)+f(y)^2+2pf(y)=a^2\iff p(f(p)+2f(y))=(a+f(y))(a-f(y)).$$Note that $a>f(y)$ and also $a\equiv \pm f(y)\pmod p.$ This forces $a\geq p-f(y)>\frac{p}{2}$ since $p$ is large and $f$ is bounded$.$ But now $$p(f(p)+2f(y))> \frac{p^2}{4}-f(y)^2\geq \frac{p^2}{4}-M^2$$$$\Rightarrow 3M\geq f(p)+2f(y)>\frac{p}{4}-M,$$which is certainly not possible$.$ Therefore $f$ is unbounded$\hspace{0.3 cm}\square.$ Now assume that $x$ is arbitrary and rewrite the condition as $$(f(y)+x)^2+xf(x)-x^2=a^2,$$where $y\in\mathbb{N}$ is such that $f(y)$ is sufficiently large (we will see how large in the following paragraph)$.$ We claim that $f(y)+x-1<a<f(y)+x+1$ for such a choice of $y.$ The first inequality rewrites as $$2f(y)>1-2x+x^2-xf(x)$$and the existence of such a $y$ is guaranteed by $f$ being unbounded$.$ Similarly$,$ the second inequality is equivalent to $$2f(y)>xf(x)-x^2-2x-1$$and again such a $y$ exists$.$ Therefore for our choice of $y,$ we must have $(f(y)+x)^2+xf(x)-x^2=(f(y)+x)^2$ and therefore $f(x)=x$ for every fixed $x$ in particular and so $$f(x)=x \hspace{0.3cm}\text{for all}\hspace{0.3cm} x\in\mathbb{N}.$$It is easily checked that this is indeed a solution$.$

The only solution is $f\equiv x$, which clearly works; we now prove that it is the only one. Let $P(x,y)$ denote the assertion. Claim: $f$ is unbounded above. Proof: Suppose the contrary, that there exists an $M\ge f(x)$ for all valid $x$. By $P(x,x)$, the expression $f(x)^2 + 3xf(x) $ is always a square, or $M^2 \ge n^2-3xf(x) = f(x)^2 > 0$ for some positive integer $n$ and every positive integer $x$. Taking $x$ large enough gives a contradiction. Claim: $f(x)\le x$ for every positive integer $x$. Proof: Suppose otherwise, that $f(x) > x$ for some $x$. Then, for that $x$ and all positive integers $y$, we must have $xf(x)+2xf(y)+f(y)^2 > (f(y)+x)^2$, or $xf(x)+2xf(y)+f(y)^2 \ge (f(y)+x+1)^2$. Expanding and simplifying the latter inequality gives $xf(x) \ge (x+1)^2 + 2f(y)$. Since $f(y)$ is unbounded above, we have a contradiction. Claim: $f(x)\ge x$ for every positive integer $x$. Proof: Suppose the contrary, that $f(x) < x$ for some $x$. Then, for said $x$ and all positive integers $y$, we have $xf(x)+2xf(y)+f(y)^2 < (f(y)+x)^2$, or $xf(x)+2xf(y)+f(y)^2 \le (f(y)+x-1)^2$. Expanding and simplifying the latter inequality gives $xf(x)+2f(y) \le (x-1)^2$, which is a contradiction since $f(y)$ is unbounded above. Now, because $f(x)\le x$ and $f(x)\ge x$, we must have $f(x)=x$ for every $x$; we are done. $\blacksquare$

$\boxed{ANSWER:f(x)=x}$

Attachments: