Let $(x,y)$ denote the assertion $xf(xy)+xyf(x)\ge f(x^2)f(y)+x^2y$. $(x,x) \implies (x^2 - f(x^2))(x - f(x)) \le 0$, denote this by $(1)$. $(0,0) \implies f(0) = 0$ $(1,1) \implies f(1) = 1$ $(x,1) \implies 2xf(x) \ge f(x^2) + x^2 \implies f(x)^2 - f(x^2) \ge (f(x) - x)^2$, denote this by $(2)$. Now, suppose that there exists $a \ge 0$ such that $a > f(a)$. $(1) \implies (a^2 - f(a^2))(a - f(a)) \le 0 \implies a^2 - f(a^2) \le 0 \implies -f(a^2) \le -a^2$ $(2) \implies (f(a) - a)^2 \le f(a)^2 - f(a^2) \le f(a)^2 - a^2 = (f(a) - a)(f(a) + a)$ As $f(a) - a < 0$, we get $f(a) - a \ge f(a) + a \implies 2a \le 0$, which isn't true. So $f(x) \ge x$ for every $x \ge 0$. That means $f(x^2) \ge x^2 \text{ } \forall x$. $(1) \implies x \ge f(x) \forall x \implies \boxed{f(x) = x \text{ } \forall x \ge 0}$ $(x<0,y<0) \implies x(yf(x) - xf(y) \ge 0 \implies yf(x) \le xf(y)$, and let that be $(3)$. $(y,x) \to (3) \implies xf(y) \le yf(x) \implies xf(y) = yf(x) \implies f(x) = cx \text{ } \forall x < 0$ $(2) \implies 2xf(x) \ge f(x^2) + x^2 \ge 2x^2 \implies x(f(x) - x) \ge 0 \implies f(x) \le x \text{ } \forall x < 0$ So, $\boxed{f(x) = cx \text{ } \forall x < 0}$ with $c \ge 1$. It's easy to see that \[ \boxed{f(x) = \begin{cases} x &\text{if } x \ge 0 \\ cx &\text{if } x < 0 \end{cases} , c \ge 1}\] works.

danepale wrote: ... $(x,x) \implies (x^2 - f(x)^2)(x - f(x)) \le 0$, denote this by $(1)$. ... $P(x,x)$ implies $(f(x^2)-x^2)(x-f(x)) \ge 0$

Yeah, I mistyped. I use the real thing later, just after I make an assumption about $a$.

danepale wrote: That means $f(x^2) \ge x^2 \text{ } \forall x$. $(1) \implies x \ge f(x) \forall x \implies \boxed{f(x) = x \text{ } \forall x \ge 0}$ I don't think that you can deduce this. Remember that in case of $f(x^2)=x^2$ then $(1)$ does not tell you anything about the relation between $f(x)$ and $x$.

I revive this in order to present my solution (I hope correct) and give an answer to this nice problem. $P(0,0)$ gives $f^2(0)\leq 0\Rightarrow f(0)=0$ $P(1,1)$ gives $(f(1)-1)^2\leq 0\Rightarrow f(1)=1$ $P(x,1)$ gives $2xf(x)\geq f(x^2)+x^2$ (1) $P(x,x)$ gives $(f(x^2)-x^2)(x-f(x))\geq 0$ (2) Suppose that there is a real number $x$ such that $f(x^2)<f^2(x)$(3).That implies $0\leq (f(x^2)-x^2)(x-f(x))<(f^2(x)-x^2)(x-f(x))=-(f(x)-x)^2(x+ f(x))\Rightarrow f(x)<-x$ From (1) we have $-2x^2>2xf(x)>f(x^2)+x^2\Rightarrow f(x^2)<-3x^2<x^2$ and hence from (2) we obtain $f(x)\geq x$ so $-x>f(x)\geq x\Rightarrow x<0$ We have hence proven that if (3) holds for a number $x$ then$x<0$ and $f(x)\geq x$. That means that for all positive $x$ we have $f(x^2)\geq f^2(x)$ so from (1) $2xf(x)\geq f(x^2)+x^2\geq f^2(x)+x^2\Rightarrow (f(x)-x)^2\leq 0 \Rightarrow f(x)=x$ for all positive $x$. Now consider $y$ to be negative.Then$P(-1,y)$ gives $-f(-y)-yf(-1)\geq f(y)+y\Rightarrow f(y)\leq -yf(-1)<0$ since $P(-1,1)$gives $f(-1)\leq -1<0$. Now if $f^2(y)>f(y^2)$ wehown that $f(y)\geq y\Rightarrow (y-f(y))\leq 0\Rightarrow (y-f(y))(y+f(y))\geq 0 \Rightarrow f^2(y)\leq y^2=f(y^2)$ a contradiction. So $f(y^2)\geq f^2(y)$ so from (1) we obtain $2yf(y)\geq f(y^2)+y^2\geq f^2(y)+y^2\Rightarrow (f(y)-y)^2\leq 0\Rightarrow f(y)=y$ for all negative $y$ and since $f(0)=0$ we have the only solution $f(x)=x$ for all real $x$

john111111 wrote: I revive this in order to present my solution (I hope correct) and give an answer to this nice problem. $P(0,0)$ gives $f^2(0)\leq 0\Rightarrow f(0)=0$ $P(1,1)$ gives $(f(1)-1)^2\leq 0\Rightarrow f(1)=1$ $P(x,1)$ gives $2xf(x)\geq f(x^2)+x^2$ (1) $P(x,x)$ gives $(f(x^2)-x^2)(x-f(x))\geq 0$ (2) Suppose that there is a real number $x$ such that $f(x^2)<f^2(x)$(3).That implies $0\leq (f(x^2)-x^2)(x-f(x))<(f^2(x)-x^2)(x-f(x))=-(f(x)-x)^2(x+ f(x))\Rightarrow f(x)<-x$ From (1) we have $-2x^2>2xf(x)>f(x^2)+x^2\Rightarrow f(x^2)<-3x^2<x^2$ and hence from (2) we obtain $f(x)\geq x$ so $-x>f(x)\geq x\Rightarrow x<0$ We have hence proven that if (3) holds for a number $x$ then$x<0$ and $f(x)\geq x$. That means that for all positive $x$ we have $f(x^2)\geq f^2(x)$ so from (1) $2xf(x)\geq f(x^2)+x^2\geq f^2(x)+x^2\Rightarrow (f(x)-x)^2\leq 0 \Rightarrow f(x)=x$ for all positive $x$. Now consider $y$ to be negative.Then$P(-1,y)$ gives $-f(-y)-yf(-1)\geq f(y)+y\Rightarrow f(y)\leq -yf(-1)<0$ since $P(-1,1)$gives $f(-1)\leq -1<0$. Now if $f^2(y)>f(y^2)$ wehown that $f(y)\geq y\Rightarrow (y-f(y))\leq 0\Rightarrow (y-f(y))(y+f(y))\geq 0 \Rightarrow f^2(y)\leq y^2=f(y^2)$ a contradiction. So $f(y^2)\geq f^2(y)$ so from (1) we obtain $2yf(y)\geq f(y^2)+y^2\geq f^2(y)+y^2\Rightarrow (f(y)-y)^2\leq 0\Rightarrow f(y)=y$ for all negative $y$ and since $f(0)=0$ we have the only solution $f(x)=x$ for all real $x$ I think this solution have a mistake (although I don't know where),because there are more solution.In fact,all function in @danepale's solution are all solution. Let $P(x,y)$ be the assertion. $P(0,0)\implies f(0)^2\le 0\implies f(0)=0$ $P(1,1)\implies (f(1)-1)^2\le 0\implies f(1)=1$ $P(x,1)\implies 2xf(x)\ge f(x^2)+x^2$ $P(x,x)\implies (f(x)-x)(x^2-f(x^2))\ge 0$ If $\exists x>0,f(x^2)>x^2$,then $f(x)\le x\implies 2x^2\ge 2xf(x)\ge f(x^2)+x^2>2x^2$,a contradiction.Hence $f(x^2)\le x^2\,\forall x>0\implies f(x)\le x\,\forall x>0$ If for some $x>0$ we have $f(x)<x$,then $x^2-f(x^2)\le 0\implies f(x^2)=x^2\implies 2x^2>2xf(x)\ge f(x^2)+x^2=2x^2$,a contradiction.Hence $f(x)=x$ for all $x>0$ If $x>0$,then $P(x,y)$ become $f(xy)\ge xf(y)$ and $P(\frac{1}{x},xy)$ become $f(y)\ge \frac{1}{x}f(xy)\implies xf(y)\ge f(xy)\implies f(xy)=xf(y)\forall x>0,y\in\mathbb{R}$ Hence,we can easily see that $f(-x)=xf(-1)\,\forall x>0\implies f(x)=cx\,\forall x<0$ where $c$ is a constant (actually,$c=-f(-1)$) Finally,by plugging in the function \[ f(x) = \begin{cases} x &\text{if } x \ge 0 \\ cx &\text{if } x < 0 \end{cases} \]and do a little casework on sign of $x$ and $y$,we get that $c\ge 1$,hence the solution is \[ \boxed{f(x) = \begin{cases} x &\text{if } x \ge 0 \\ cx &\text{if } x < 0 \end{cases} , c \ge 1}\]which satisfied the condition.

john111111 wrote: If $x>0$,then $P(x,y)$ become $f(xy)\ge xf(y)$ ?

Taking $(x,y)=(0,0)$ and $(x,y)=(1,1)$, we derive $f(0)=0$ and $f(1)=1$. Taking $y=1$ gives $$2xf(x)\ge f(x^2)+x^2.$$Note that this gives $f(-1)\le -1$. Letting $y=x$ yields $0\ge (f(x^2)-x^2)(f(x)-x)$. If $x\ge 0$ and $f(x^2)\ge x^2$ for all $x\ge 0$, we may multiply both sides by $2x$ to obtain $$0\ge (f(x^2)-x^2)(2xf(x)-2x^2)\ge (f(x^2)-x^2)^2.$$Thus, $f(x^2)=x^2$, which implies $f(x)=x$ for all $x\ge 0$. Otherwise, we have $f(x^2)<x^2$ for some $x$. Thus, $f(x)\ge x$. Note that we also have $0\ge (f(x^4)-x^4)(f(x^2)-x^2)$, so we must have $f(x^4)\ge x^4$. Replacing $x$ with $x^2$, and taking $y=1$ in the original equation gives $$2x^2f(x^2)\ge f(x^4)+x^4\ge 2x^4.$$Hence, $f(x^2)\ge x^2$, which contradicts the assumption that $f(x^2)<x^2$. So, $f(x)=x$ for all $x\ge 0$. Swapping the sign on both $x$ and $y$ in the original equation (and assuming $x,y$ are positive) gives $-xf(xy)+xyf(-x)\ge f(x^2)f(-y)-x^2y$. Therefore, $f(-x)/x\ge f(-y)/y$. But swapping $x$ and $y$, we see that $f(-x)/x=f(-y)/y$ for all positive $x,y$. Hence, $f(x)=cx$ for some $c$ for all $x< 0$. Observe that $-2xf(-x)\ge f(x^2)+x^2=2x^2$, so we have $f(-x)/x\le -1$. This gives, $f(x)=cx$ for some $c\ge 1$ for all $x< 0$. Thus, the solution is \[ \boxed{f = \begin{cases} x &\text{if } x \ge 0 \\ cx &\text{if } x < 0 \end{cases} , c \ge 1},\]and it is easy to check it works.

winnertakeover wrote: $$2xf(x)\ge f(x^2)+x^2.$$Letting $y=x$ yields $0\ge (f(x^2)-x^2)(f(x)-x)$. If $x\ge 0$, we may multiply both sides by $2x$ to obtain $$0\ge (f(x^2)-x^2)(2xf(x)-2x^2)\ge (f(x^2)-x^2)^2.$$ If $f(x^2)<x^2,$ then $$(f(x^2)-x^2)(2xf(x)-2x^2)\le (f(x^2)-x^2)^2$$

Ahh, yes you are right. I will fix my post later. EDIT: fixed

\[ \boxed{f(x) = \begin{cases} x &\text{if } x \ge 0 \\ cx &\text{if } x < 0 \end{cases} , c \ge 1}\]

Let $P(x,y)$ denote the given assertion. We claim that all such functions for $x\geqslant 0$ are $f(x)=x$ and $f(-x)=-ax$ where $a\geqslant 1.$ These clearly satisfy. Note that $f(0)=0$ and $f(1)=1$ by $P(0,0)$ and $P(1,1).$ Consider a non-negative number $z.$ A. $f(z^2)>z^2:$ Then $P(z,z)$ implies $f(z)\leqslant z.$ And $P(z,1)$ implies $2z^2<f(z^2)+z^2\leqslant 2zf(z)\leqslant 2z^2$ a contradiction. B. $f(z^2)<z^2:$ Then $P(z,z)$ implies $f(z)\geqslant z$ and $P(z^2,z^2)$ implies $f(z^4)\geqslant f(z^4).$ Again $P(z^2,1)$ implies $2z^4>2z^2f(z^2)\geqslant f(z^4)+z^4\geqslant 2z^4$ a contradiction. Therefore $f(z^2)=z^2$ means $f(z)=z.$ Now pick $u,v<0.$ Combining $P(u,v)$ and $P(v,u)$ implies $f(u)/u=f(v)/v=a.$ Finally $P(u,1)$ implies $f(u)\leq u.$ Hence $a\geq 1$ as desired.