Could you please post your full solution? It seem that your solution is totally different from mine

The constant solutions are $P(x)=0$ and $P(x)=1$. From now on we assume that the degree of $P$ is at least one. (1) $P$ does not have any real zero. For this we note that $P(a)=0$ implies $P(a^2-a+3)=0$, and that $a^2-a+3>a$. Hence any real zero $a$ would yield an infinite strictly increasing sequence of other real zeroes; this would force the polynomial $P$ to be constant. (2) By setting $x=1-y$, we get $P(1-y)P(2-y)=P(y^2-y+3)$. This implies $P(1-x)P(2-x)=P(x)P(x+1)$ for all real $x$. (3) From (2) with $x=0$ we get $P(1)P(2)=P(0)P(1)$. Together with (1), this yields $P(2)=P(0)$. By induction on $n\ge2$, one easily shows while using (1) and (2) that $P(n)=P(2-n)$. This implies $P(x)=P(2-x)$ for all real $x$.

$P = 0$ is apparently a solution. So we only have to work on the case $P\neq 0$. Claim 1. $P$ has no real roots.

Claim 2. $P(x) = (x^2-2x+3)^n$ for a $n\in\mathbb{N}_0$

In conclusion, $P(x) = (x^2-2x+3)^n$ and $P(x) = 0$ are the only solutions. Remark. Finding a special solution might be the most indirect part in this solution. However, if one looks carefully at the roots of $P$ (and maybe the fixed points of $x^2-x+3$), then trying to plug in $P(x) = x^2-2x+3$ is very natural. Once $x^2-2x+3$ is verified to be a solution, it follows immediately that $(x^2-2x+3)^n$ are also solutions. In addition, one can also plug in $P(x) = ax^2+bx+c$ to solve $a,b,c$.

USJL wrote: Claim 2. $P(x) = (x^2-2x+3)^n$ for a $n\in\mathbb{N}_0$

I don't understand. If you plugging in $P(x) = Q(x)+ f(x) $ to the equation, you must have $Q(x).Q(x+1)+Q(x).f(x+1)+f(x).Q(x+1)+f(x).f(x+1)=f(x^2-x+3)+Q(x^2-x+3)$ then how you do to get $f(x)Q(x+1)+f(x+1)Q(x) = Q(x^2-x+3)$

hieudg wrote: USJL wrote: Claim 2. $P(x) = (x^2-2x+3)^n$ for a $n\in\mathbb{N}_0$

I don't understand. If you plugging in $P(x) = Q(x)+ f(x) $ to the equation, you must have $Q(x).Q(x+1)+Q(x).f(x+1)+f(x).Q(x+1)+f(x).f(x+1)=f(x^2-x+3)+Q(x^2-x+3)$ then how you do to get $f(x)Q(x+1)+f(x+1)Q(x) = Q(x^2-x+3)$ There was a typo in my solution (fixed), but it doesn't make the argument go wrong. Thanks for pointing out my mistake by the way

The answer is $P(x)=(x^2-2x+3)^n$. It clearly works. Let $S$ be the set of roots of $P$. We know if $r\in S$ then $P(r^2-r+3)=0$, so o$r^2-r+3=0$. Also, $P((r-1)^2-(r-1)+3)=P(r-1)P(r)=0$ so $P(r^2-3r+5)=0$. This means $r^2-r+3\in S$ and $r^2-3r+5\in S$. Note $S$ is finite. Claim 1: $P\cap \mathbb{R}=\emptyset$ Let $x$ be the maximum real in $S$. Then note $x^2-x+3>x$ by AM-GM. Claim 2: If $a+bi \in S$ then $a=1$. Proof: Consider any $r=a+bi$. Then $Im((r-\frac 12)^2)=2(a-\frac 12)b$ and $Im((r-\frac 32)^2)=2(a-\frac 32)b$. By triangular inequality, $\max \{|2a-1|, |2a-3|\} \ge 1$. Therefore, if $a\ne 1$ then I can always generate elements of $S$ with arbitrarily large imaginary values. Therefore. at some point, $a=1$ and $Re((1+bi-\frac 12)^2 + \frac{11}{4})=1$ implies $b=\sqrt{2}$. Therefore, $\{1+\sqrt{2}i, 1-\sqrt{2}i\} \subseteq S$. We can check $(1+\sqrt{2}i)^2- (1+\sqrt{2}i)+3=1+\sqrt{2}i$ and $(1+\sqrt{2}i)^2 - 3(1+\sqrt{2}i) + 5 = 1-\sqrt{2}i$. Therefore, if I use this method of generating roots (elements of $S$), I must stop when I see these two elements. In other words, if $r\in S$, then if I apply $f(r)=r^2-r+3$ and $g(r)=r^2-3r+5$, I must bump into one of the two. Suppose $f(r)=1+\sqrt{2}i$, then we can see $r=1+\sqrt{2}i$ or $r=-\sqrt{2}i$. However, $g(-\sqrt{2}i)=3+3\sqrt{2}i$. We have $|g(-\sqrt{2}i)|=3\sqrt{3}$. If $|r|\ge 3\sqrt{3}$ then $|f(r)| \ge |r|^2-|r|-3\ge 18$. We can show $|f^k(r)| > |f^{k-1}(r)|$ and $f^k(r)\in S$ for all $k\in \mathbb{N}$, contradicting that $S$ is a finite set. Therefore $\pm \sqrt{2}i \notin S$. This means for a number to reach one of $\{1+\sqrt{2}i, 1-\sqrt{2}i\}$ in $n$ steps, it must reach it in $n-1$ steps, so it must reach it in 0 steps, implying $S=\{1+\sqrt{2}i, 1-\sqrt{2}i\}$

Consider the imaginary part of the roots.

If \( P \) is constant, then \( P \equiv 0 \) or \( P \equiv 1 \). Assume \( P \) is non-constant. Note that the leading coefficient \( a_n \) of \( P(x) \) is \( 1 \), since \( a_n^2 = a_n \) and \( a_n \neq 0 \). Let \(\deg(P(x)) = n\). We claim the only solution is \( (x^2 - 2x + 3)^n \), which clearly works. Assume there is another solution for \( P(x) \), for the sake of contradiction. Let \[ Q(x) = P(x) - (x^2 - 2x + 3)^n, \]and clearly \(\deg(Q(x)) < n\) (since the terms with an exponent of \( n \) cancel out). Substitute this into our equation to obtain: \[ (Q(x) + (x^2 - 2x + 3)^n)(Q(x + 1) + (x^2 + 2)^n) = Q(x^2 - x + 3) + (x^4 - 2x^3 + 5x^2 - 4x + 6)^n. \]Expanding and rearranging terms: \[ Q(x)(x^2 + 2)^n + \\ \\ Q(x)Q(x + 1) \\ + \\ \\ Q(x + 1)(x^2 - 2x + 3)^n = Q(x^2 - x + 3). \]Since $\deg(Q(x)) < n$, the degree of the LHS is $n + \deg(q)$, while the degree of the RHS is $2\deg(Q)$, but $n + \deg(q) > 2\deg(Q)$, so we have a contradiction.