Let $Q(x)$ have a degree $a_1$. Let $P(x)$ have degree $b_1$. Then, $$3ab = 1 + 3a_1 + b_1 \implies (3a_1-1)(b_1-1) = 2 \implies a_1=1,b_1=2$$Therefore, $Q$ is linear and $P$ is a quadratic. Let $P(x) = a(x-b)(x-c)$ and $Q(x) = x-d$. Plug in $x=d$ in the original equation to get: $P(0) = 0$. So instead, let’s write $P(x) = ax(x-b)$ and $Q(x) = x-d$. Therefore, $$a(x-d)^3((x-d)^3-b) = ax^2(x-b)(x-d)^3 \implies ((x-d)^3-b) = x^2(x-b) \implies x^3-3x^2d+3xd^2-d^3-b=x^3-x^2b$$Hence, $d = b= 0$ and $P(x) = ax^2$ and $Q(x) = x$ and these are the only solutions.

Belarus 2019 https://artofproblemsolving.com/community/c6h1907950p13060758

socrates wrote: Belarus 2019 https://artofproblemsolving.com/community/c6h1907950p13060758 Yes, I recalled having solved a problem of similar flavor before, but I didn’t remember where from.

@above im pretty sure it is a bad practice to use the variable a twice in ur proof.

bever209 wrote: @above im pretty sure it is a bad practice to use the variable a twice in ur proof. Yes, it is bad practice. I have fixed the mistake now.

So let's rewrite our polynomials as: $$P(x) = \sum_{i=0}^{n} a_ix^i$$$$Q(x)=\sum_{i=0}^{m} b_ix^i $$So now we know that the degrees must be equal we have that: $$ 3mn=1+n+3m$$Solving this in $\mathbb{N}$ we get that $m=1$, and $n=2$. But before we plug our new definitions for our polynomials, let's examine the equation when $x=0$. For sake of saving space we denote $a_0 = Q(0)$, so now we have that: $P(a_0) = 0$, now plugging in $x=a_0$, we get that: $P(Q(a_0)^3)=0$, and we can go on and on,... So obviously we don't have that $P(x)$ doesn't have an infinite number of zeroes. So we see that in our most extreme case we have that: $$ Q(Q(a_0)^3)^3=a_0^3$$. Solving that where, $Q(x)=a_1x+a_0$, we get that $a_0=0$ So now we have that $Q(x)=a_1x$, and because $P(0)=0$, we have that $P(x)=b_2x^2+b_1x$. So now plugging all that back in we get the following: $$ b_2((a_1x)^3)^2 + b_1(a_1x)^3= x(b_2x^2+b_1x)(a_1x)^3 $$$$ b_2a_1^3x^3 + b_1 = b_2x^3 + b_1x^2 $$So from here we have that $b_2a_1^3=b_2 \iff a_1^3=1 \iff a_1 = 1$ so we see that $b_2$ can be anything we want, and from that we have $b_1 = b_1x^2 \iff b_1 = 0$, we have that $b_1=0$ since the second relation must hold $\forall x \in \mathbb{R}$. So the solutions are $Q(x)=x$ and $P(x)=b_2x^2$, $\forall b_2 \in \mathbb{R}$ . . .

very nice problem! Claim:- $P(x)= ax^2$ $\forall$ $a \in \mathbb{R^{*}}$ , $Q(x)=x$ are the only polynomial satisfying the problem statement Proof:-denote $\text{Deg(P(x))}=m, \text{Deg(Q(x))}=n$ so we get $3mn=3n+m+1 \implies (3n-1)(m-1)=2\implies n=1, m=2$ so we set $P(x)=ax^2+b$ and $Q(x)=jx+k$ where $a,j \in \mathbb{R^{*}}$ and $b,k \in \mathbb{R}$ now we notice $a(jx+k)^{6}+b=x(ax^2+b)(jx+k)^3$ now plugging $x=0$ we get: $ak^{6}+b=0 $ also comparing coefficient of $x^{6}$ we get that $aj^{6}=aj^{3}\implies j=1$ also comparing coefficient of $x$ we get that $ak^{5}j=k^3b \implies ak^{5}=k^{3}b$ Case1:- $k=0$ clearly, this gives $b=0$ which gives $P(x)=ax^2$ and $Q(x)=x$ Case2:- $k\neq 0$ then we have $ak^2=b$ but we also had $ak^{6}=-b$ so we get $k^{4}=-1$ which is clearly not possible hence we get $\boxed{P(x)=ax^2 \hspace{0.2cm} \forall \hspace{0.2cm} a \in \mathbb{R^{*}}, Q(x)=x}$ as the only solutions , as claim follows $\blacksquare$

Here's a bit of a lengthy solution First of all let $P(x)=\sum_{i=0}^{n}a_ix^i\text{ and }Q(x)=\sum_{i=0}^{k}b_ix^i$ thus $\deg P=n\text{ and }\deg Q=k$ so $\deg Q^3=3k$ and thus $\deg P(Q^3)=3nk$ Furthermore $P(x)Q(x)^3=R(x)=\sum_{i=0}^{n+3k}\pi_i x^i$ therefore $\deg R=n+3k$ and thus $\deg \text{RHS}=1+n+3k$ So $\deg\text{LHS}=\deg\text{RHS}\Longleftrightarrow 3nk=1+n+3k\Longrightarrow (3k-1)(n-1)=2$ now checking the factors yields $n=2\text{ and }k=1$ Thus $P(x)=ax^2+bx+c\text{ and }Q(x)=dx+e$ Now letting $x\to-e/d$ yields $Q(-e/d)=0$ thus $P(Q(-e/d)^3)=-\frac{e}{d}P(-e/d)Q(-e/d)^3\Longrightarrow P(0)=0$ therefore $c=0$ and $P$ can be rewritten as $P(x)=x(ax+b)$ So furthermore let $x\to0$ yields $Q(0)=e$ thus $P(e^3)=0\Longrightarrow e^3(ae^3+b)=0$ thus either $e=0\text{ or }ae^3+b=0$ Case 1: $e=0$ In this case the polynomials become $P(x)=x(ax+b)\text{ and }Q(x)=dx$ Using these in our original equation yields $P(d^3x^3)=d^3x^5(ax+b)\Longrightarrow d^3x^3(ad^3x^3+b)=d^3x^5(ax+b)$ Furthermore notice that letting $x\to-\frac{b}{a}$ gives us $-d^3\cdot\frac{b^3}{a^3}\left(b-\frac{d^3b^3}{a^2}\right)=0$ so either $b=0\text{ or }b-\frac{d^3b^3}{a^2}=0$ Subcase 1.1: $b=0$ So the polynomials are equivalent to $P(x)=ax^2\text{ and }Q(x)=dx$ Plugging these into our original equation yields $P(d^3x^3)=a\cdot d^3\cdot x^6\Longrightarrow a\cdot d^6\cdot x^6=a\cdot d^3\cdot x^6\Longleftrightarrow x^6\cdot d^6=x^6\cdot d^3$ thus $d=1$ and $Q(x)=x$ Therefore $\boxed{P(x)=ax^2\text{ and }Q(x)=x}$ is a solution. Subcase 1.2: $b-\frac{d^3b^3}{a^2}=0$ $\text{FTSOC}$ assume that $b\neq0$ so the condition can be rewritten as $d=\left(\frac{a}{b}\right)^{2/3}$ and $Q(x)=\left(\frac{a}{b}\right)^{2/3}x$ Furthermore plugging this into our original equation yields $ax^3(a^2-b)=b^2(x^2-1)$ now letting $x\to1$ yields $a(a^2-b)=0$ thus $a^2=b$ which forces $Q(x)=a^{-2/3}x$ Moreover letting $x\to a$ gives us $Q(a)=a^{1/3}\text{ and }P(a)=a(a^2+b)$ which is equivalent to $(a^2+b)(a^2-a)=0$ after expansion in our original equation. So $a=\pm 1\text{ or }b=-a^2$

If $b=-a^2$ we have that $b=-a^2=a^2$ which forces $a=0$ which is clearly a contradiction as $P\not\equiv c$ by the problem statement. So we get a contradiction in both options, so our assumption must've been false. Thus $b$ must equal $0$ and we refer to Subcase 1.1 Case 2: $ae^3+b=0\Longleftrightarrow b=-ae^3$ So we have that $P(x)=ax(x-e^3)\text{ and }Q(x)=dx+e$ Now we have that $P((dx+e)^3)=ax^2(x-e^3)(dx+e)^3$ Furthermore let $x\to e^3$, using this yields $P(e^3(de^2+1)^3)=0$ which is equivalent to $ae^6(de^2+1)^3((de^2+1)^3-1)=0$ so $e=0\text{ or }(de^2+1)^3-1=0\text{ or }de^2+1=0$ Subcase 2.1: $e=0$ In this subcase refer back to Case 1. Subcase 2.2: $(de^2+1)^3-1=0$ This is equivalent to $(de^2+1)^3=1\Longrightarrow de^2+1=1\text{ so }de^2=0\Longleftrightarrow e=0$, so also refer back to Case 1. Subcase 2.3: $de^2+1=0$ We have that $P((dx+e)^3)=ax^2(x-e^3)(dx+e^3)\Longrightarrow (dx+e)^3((dx+e)^3-e^3-x^2(x-e^3))=0$ Now let $x\to1$ using this we obtain $(d+e)^3((d+e)^3+1)=0$ so either $d+e=0\text{ or }(d+e)^3+1=0$ Sub-subcase 2.3.1: $d+e=0\Longleftrightarrow d=-e$ Using $de^2+1=0$ we have that $-e^3+1=0\Longrightarrow e=1$ which forces $d=-1$ so $P(x)=ax(x-1)\text{ and }Q(x)=1-x$ Furthermore let $x\to{-1}$ yields $Q(-q)=2\text{ and }P(-1)=2a$ Therefore we have that $P(Q(-1)^3)=-P(-1)Q(-1)^3\Longrightarrow P(8)=-16a\text{ thus }56a=-16a$ which forces $a=0$ which is a contradiction. Sub-subcase 2.3.2: $(d+e)^3+1=0\Longleftrightarrow d=-(e+1)$ Using $de^2+1=0$ yields $e^2+e-1$ so $e_{1/2}=\frac{-1\pm\sqrt{5}}{2}$ thus $d=-\left(\frac{-1\pm\sqrt{5}}{2}+1\right)$ Furthermore $Q(x)=\left(-\frac{-1\pm\sqrt{5}}{2}-1\right)x+\frac{-1\pm\sqrt{5}}{2}$ so $Q(1)=-1$ Now letting $x\to1$ yields $P(Q(1)^3)=P(1)Q(1)^3\Longleftrightarrow P(-1)=-P(1)$ $\therefore a\left(1+\left(\frac{-1\pm\sqrt{5}}{2}\right)^3\right)=a\left(1-\left(\frac{-1\pm\sqrt{5}}{2}\right)^3\right)\Longleftrightarrow\left(\frac{-1\pm\sqrt{5}}{2}\right)^3=-\left(\frac{-1\pm\sqrt{5}}{2}\right)^3$ which is clearly a contradiction. So in conclusion $\boxed{P(x)=ax^2\text{ and }Q(x)=x\text{ are the only polynomials that satisfy the problem}}$ $\blacksquare$.

$\color{red} \textbf{Claim:-}$ The only solution works that is $\boxed{P(x)=ax^2}$ $\forall a \in \mathbb{R^{*}}$ and $\boxed{Q(x)=x.}$ $\color{blue} \textbf{Proof:-}$ Assuming $P(x)=a_nx^n+a_{n-1}x^{n-1}+.....+a_1x+a_0$ be a $n$ degree polynomial and assuming $Q(x)=b_mx^m+b_{m-1}x^{m-1}+...+b_1x+b_0$ be a $m$ degree polynomial. Let's look at the degree we get, $\boxed{\text{deg of P(x) is n.}}$ and $\boxed{\text{deg of Q(x) is m.}}$ satisfying $$P((Q(x))^3)=xP(x)(Q(x))^3\implies 3mn=1+n+3m\implies(3m-1)(n-1)=2\implies n=2,m=1$$Therefore the only solution works that is $P(x)=ax^2+b.$ and $Q(x)=dx+e.$ Now assume $P(x)=ax^2+b$ and $Q(x)=dx+e$ satisfying the condition we get, $$P(Q(x)^3)=x\cdot P(x)(Q(x)^3)\implies a(dx+e)^6+b=x(ax^2+b)(dx+e)^3$$Plugging $x=0$ we get, $ae^6+b=0.$ and comparing coefficients of $x^6$ we get, $ad^6=ad^3\implies d=1.$ and comparing coefficient of $x$ we get that $ae^{5}d=e^3b \implies ae^{5}=e^{3}b.$ From this there are two cases arise when $ \textbf{Case-1:-}$ $\boxed{e=0}$ we get, $b=0.$ which gives the polynomial as $\boxed{P(x)=ax^2.}$ and $\boxed{Q(x)=x.}$ $ \textbf{Case-2:-}$ $\boxed{e\neq 0}$ from this we get, $ae^2=b$ but we also had $ae^5=-b$ therefore we get, $e^4=-1$ which is clearly not possible. Hence the only possible polynomials is $\boxed{P(x)=ax^2 \hspace{0.2cm} \forall \hspace{0.2cm} a \in \mathbb{R^{*}}, Q(x)=x}$ and the claim follows.

The degree of $Q(x)^3$ is $3n$, so that of $P(Q(x)^3)$ is $3mn$, while that of the right-hand side of the given equation is $3n + m + 1$, thus $3mn = 3n + m + 1$. Hence $3n(m-1) = m+1$, so $m-1$ divides $m+1 = (m-1) + 2$, so either $m-1 = 2$ (i.e. $m=3$, which does not work) or $m-1 = 1$, i.e. $m=2$ and $n=1$. Hence $P(x) = ax^2 + bx + c$ and $Q(x) = rx + s$ for some real numbers $a\neq 0$, $b$, $c$, $r\neq 0$ and $s$. Setting $x=-\frac{s}{r}$ yields $c = P(0) = 0$. Hence $P(Q(x)^3) = (rx+s)^3(a(rx+s)^3 + b)$ and $xP(x)Q(x)^3 = x^2(ax+b)(rx+s)^3$ and now cancelling $(rx+s)^3$ leads to \[ ar^3x^3 + 3ar^2sx^2 + 3ars^2x + as^3 + b = ax^3 + bx^2. \]The latter holds if and only if $ar^3 = a$, $3ar^2s = b$, $3ars^2 = 0$ and $as^3 + b = 0$. The first one gives $r=1$, the third one (as $a\neq 0$) gives $s=0$, and now the second and fourth one hold only for $b=0$. In conclusion, all non-constant solutions are$P(x) = ax^2$ with $Q(x) = x$, where $a\neq 0$ is an arbitrary real number.

Let $\deg P=m, \deg Q=n$ we have $3mn=3n+m+1$ so $m=2$ and $n=1$. Take the root $r$ of $Q$, plug in $x=r$ we have $P(Q(r)^3)=0$ so $P(0)=0$. Take $P(x)=ax^2+bx, Q(x)=cx+d$. We have $aQ(x)^6+bQ(x)^3=x(ax^2+bx)Q(x)^3$ so we have $a(cx+d)^3+b=x(ax^2+bx)$. as $ac^3=a$ we have $c=1$ and as $3ad^2x=0$ we have $d=0$ and as $3ad=b$ we have $d=0$ Therefore $P(x)=ax^2$ and $Q(x)=x$.