https://aops.com/community/p11377651 rmtf1111 wrote: First I will prove that $(xy+1)^2+\frac{(x+y+4)^2}{3} \le (x^2+3)(y^2+3) \Longleftrightarrow$ $$3x^2+3y^2+8 \ge 2xy+\frac{(x+y+4)^2}{3} \Longleftrightarrow$$$$\frac{8}{3}(x+y)^2+8\ge 8xy+\frac{8}{3}(x+y)+\frac{16}{3} \Longleftrightarrow$$$$(x+y)^2+1\ge 3xy+x+y \Longleftrightarrow \frac{3}{4}(x-y)^2+\left(\frac{x+1}{2}-1\right)^2 \ge 0$$Now, by C-S we have that $$(xyz+x+y+z+4)^2=(z(xy+1)+(x+y+4))^2 \le (z^2+3)\left( (xy+1)^2+\frac{(x+y+4)^2}{3}\right) \le (x^2+3)(y^2+3)(z^2+3)$$ https://aops.com/community/p11389621 Bobson wrote: It's just Hadamard inequality for $\begin{pmatrix}x & \Phi & -\phi\\-\phi & y & \Phi\\\Phi & -\phi & z\end{pmatrix}$, where $\Phi$ and $-\phi$ are roots of $r^2=r+1$.

Let $x+y+z=p,xy+yz+za=q,xyz=r$. The given becomes $r(8p+8)\leq 8p^2+3q^2-18q+11-8p$ fix $p,q$. Then $r(8p+8)$ is a linear function of $r$ so it maximizes when $r$ takes minimum or maximum value. In any case it's enough to take $r=0$ or WLOG $x=y$ Let $z=0$. Then we have to show that $x^2(8+3y^2)-x(2y+8)+8y^2-8y+11\geq 0$ It suffice to show that $4(y+4)^2-4(8y^2-8y+11)(8+3y^2)\leq 0\Leftrightarrow ...$ $\Leftrightarrow y^4-y^3+4y^2-6y+3\geq 0 \Leftrightarrow y^2(y^2-y+1)+3(y-1)^2\geq 0 $ which is true. Now let $x=y$ We have to show that $z^2(x^2+3)^2+3(x^2+3)^2\geq z^2(x^2+1)^2+(2x+4)^4+2z(x^2+1)(2x+4)$ It suffice to show that $(4x^3+4x+8x^2+8)^2-4(4x^2+8)(3x^4+14x^2-16x+11)\leq 0 \Leftrightarrow .. $ $..\Leftrightarrow (x-1)^2(x^2+3)\geq 0$ which is true.