geo0420 wrote: Let $deg\;p =m, deg\;q=n.$ Check some simple cases when $m, n$ are small and we get $(a, b)=(-1, -1).$ When $m, n$ are sufficiently big, $m=n$ and comparing coefficients give us $p(x)=rq(x)$ for some real $r,$ which leads to contradiction. Do you think the condition is for all $p$ and $q$? However, actually the question is that there exists $p$ and $q$.

Very easy problem. If we put two root of equation of $x^2 -x-1$, we get $p(x^2)q(x+1)-p(x+1)q(x^2)=0=x^2+ax+b$, so $x^2 -x-1 =x^2+ax+b, a=b=-1$. Q.E.D.

@above Nice solution!! tobylong wrote: Do you think the condition is for all $p$ and $q$? However, actually the question is that there exists $p$ and $q$. I just tried to find all tuples $(p(x), q(x), a, b)$ which satisfies the condition, and concluded that for all such tuples, $(a, b)=(-1, -1).$

Exact same solution as #5 but posting for storage. Let $z$ be a root of the equation $z^2 = z+1$, note that this is real since $z = \frac{1 \pm \sqrt{5}}{2}$. Putting $x=z$, we get that $z^2 + az + b = p(z^2)q(z+1) - p(z+1)q(x^2) = p(z^2)q(z^2) - p(z^2)q(z^2) = 0$. So, $z^2 + az + b = 0$ but since here, the leading coefficient is $1$, there is only one possible value for $a,b$, which is the same as that in the equation $z^2 - z - 1 = 0$, $(a,b) = (-1,-1)$ So, the only solution is $(a,b) = (-1,-1)$

Plugging $x=0$ and $x=1$ and adding these two, we get $a=2b+1$. Let $\alpha$ be one of the roots of the quadratic $x^2 - x - 1 = 0$. It is clear that $\alpha \in \mathbb{R}$. Plugging $x=\alpha$, we get $0=P(\alpha^2)Q(\alpha + 1) - P(\alpha + 1)Q(\alpha^2) = \alpha^2 + a\alpha + b \implies \alpha + 1 + a\alpha + b = 0$. Using $a=2b+1$, we get $2\alpha(b+1) = -(b+1) \implies b=-1 \implies a=-1$. To see that $(a,b)=(-1,-1)$ work, just take $Q(x)=1$ and $P(x)=x$.