Let $P(x,y)$ implies $f(x + yf(x + y)) = y^2 + f(xf(y + 1))$
1) $f(1)=1$ and $f(0)=0$
$P(x,0) \Longrightarrow f(x)=f(xf(1))$
$P(xf(x), 0) \Longrightarrow f(xf(x))=f(xf(x)f(1))$
$P(0,x) \Longrightarrow f(xf(x))=x^2+f(0)$
$P(0,xf(1)) \Longrightarrow f(xf(x))=f(xf(x)f(1))=f(xf(1)f(xf(1)))=x^2f(1)^2+f(0)$
$\Longrightarrow f(xf(x))=x^2+f(0)=x^2f(1)^2+f(0)$
$\Longrightarrow x^2=x^2f(1)^2$
So $f(1)=1$ or $f(1)=-1, \forall x \ne 0$
Case 1: $f(1)=-1$
$P(x,0) \Longrightarrow f(x)=f(-x)$
$P(1,0) \Longrightarrow f(-1)=-1$
$P(1,-2) \Longrightarrow f(3)=3$
$P(-1,2) \Longrightarrow f(3)=f(-3)=7=3$ contradiction.
Case 2: $f(1)=1$
$P(0,1) \Longrightarrow f(0)=0$
So $f(1)=1$ also true when $x=0$
2) $f(x)=1 \Longleftrightarrow x=1$
If exist $a : f(a)=1$
$P(0,a) \Longrightarrow a^2=1$
$\Longrightarrow a=1$ or $a=-1$
If $a=-1$, it implies $f(-1)=1$
$P(1;-2) \Longrightarrow 1=5$ contracdiction.
So $f(x)=1 \Longleftrightarrow x=1$
3) $f(x)=x, \forall x \in \mathbb{R}$
$P(x+1,-1) \Longrightarrow f(x+1-f(x))=1$
Since $f(x)=1 \Longleftrightarrow x=1$, it implies $x+1-f(x)=1$
And so $\boxed{f(x)=x, \forall x \in \mathbb{R}}$
Thank you Dear mathlinks.