Ovchinnikov Denis wrote:
Found all functions $f: \mathbb{R} \to \mathbb{R}$, such that for any $x,y \in \mathbb{R}$ holds next equality:
$f(x^2+xy+f(y))=f^2(x)+xf(y)+y$.
Let $P(x,y)$ be the assertion $f(x^2+xy+f(y))=f^2(x)+xf(y)+y$
1) $f(0)=0$
===========
Let $f(0)=a$
(a) : $P(4a,0)$ $\implies$ $f(16a^2+a)=f^2(4a)+4a^2$
(b) : $P(-4a,0)$ $\implies$ $f(16a^2+a)=f^2(-4a)-4a^2$
(c) : $P(-4a,4a)$ $\implies$ $f(f(4a))=f^2(-4a)-4af(4a)+4a$
(d) : $P(0,4a)$ $\implies$ $f(f(4a))=4a+a^2$
(a)-(b)+(c)-(d) : $0=(f(4a)-2a)^2+3a^2$ and so $a=0$
Q.E.D.
2) $f(x)=x$
===========
(a) : $P(0,x)$ $\implies$ $f(f(x))=x$ and so $f(x)$ is a bijection
(b) : $P(-x,x)$ $\implies$ $f(f(x))=f^2(-x)-xf(x)+x$
(c) : $P(x,0)$ $\implies$ $f(x^2)=f^2(x)$
(d) : $P(-x,0)$ $\implies$ $f(x^2)=f^2(-x)$
-(a)+(b)+(c)-(d) : $0=f(x)(f(x)-x)$
Since $f(0)=0$ and $f(x)$ is a bijection, we have $f(x)\ne 0$ $\forall x\ne 0$ and the above equality becomes $f(x)=x$ $\forall x\ne 0$
And so $f(x)=x$ $\forall x$ which indeed is a solution.