Suppose the three roots of $Q$ are $r,s,t$. We have that $r+s+t=0$, $rs+st+rt=-21$, and $rst=-35$ by Vieta's. If the roots of $Q$ cyclically permute in the polynomials $P(x)=x^3+ax+b$, then we have that $$r^3+ar+b=s$$ $$s^3+as+b=t$$ $$t^3+at+b=r$$ Adding up our three equations and noting that $r+s+t=0$ from above, we get that $$-r^3-s^3-t^3=3b$$ By Newton's Identities we have $P_1=0$, $P_2=42$ and $P_3=-105$. This implies that $3b=105$. Dividing both sides by $3$ gives us $\boxed{b=35}$. Looking back at our original polynomial, note that we can rewrite it as $x^3+35=21x$, meaning that $r^3+35=21r$, $s^3+35=21s$ and $t^3+35=21t$ since $r,s,t$ are roots. Thus we have that our cyclic equations turn into $$t(21+a)=r$$ $$r(21+a)=s$$ $$s(21+a)=t$$ Substituting the first into the second and the new second into the third, we get $$t(21+a)^3=t$$ Clearly $t\neq 0$ since $0$ is not a root of $x^3-21x+35$. Dividing by $t$ on both sides gives us $(21+a)^3=1$. Obviously the only real root of this is $-20$ (you can find others with roots of unity) so we have $\boxed{a=-20}$.

The problem actually says $P$ is a quadratic polynomial, not cubic. In this case we may easily check that $b = -14$ through a similar method to above, and then we may easily obtain that $a = 2$ through some other manipulations, thus $P(x) = x^2 + 2x - 14$. In order to verify that it works, we check that $Q(x)$ divides $Q(P(x))$, and hence $P(r), P(s)$, and $P(t)$ are also roots of $x$. After checking that none of the roots of the polynomial $P(x) - x$ are roots of $Q$, it follows that $P(r) \neq r, P(s) \neq s$ and $P(t) \neq t$. Finally, we may verify that $P(r), P(s), P(t)$ are all distinct. These three facts together imply that $P$ indeed permutes the roots of $Q$.

I apologize for the typo; indeed $P$ is supposed to be a quadratic polynomial.

From Vieta relations we get: {1} $r+s+t=0$ {2} $rs+st+tr=-21$ {3} $rst=-35$ 1) $0=(r+s+t)^2=r^2+s^2+t^2+2(rs+st+tr)$ {4} $\Rightarrow r^2+s^2+t^2=42$ 2) $r^3+s^3+t^3-3rst=(r+s+t)(r^2+s^2+t^2-rs-st-tr)=0$ {5} $\Rightarrow r^3+s^3+t^3=3rst=-105$ We want that: $r^2+ar+b=s\longrightarrow r^3+ar^2+b=sr$ $s^2+as+b=t\longrightarrow s^3+as^2+b=st$ $t^2+at+b=r\longrightarrow t^3+at^2+b=rt$ Adding the left-column equations we get: $\{4\}+a\{1\}+3b=\{1\} \longrightarrow 42=-3b \Rightarrow b=-14$ Adding the right-column we get: $\{5\}+a\{4\}+3b=\{2\} \longrightarrow 42a=84 \Rightarrow a=-2$ $Q.E.D$ (The checking is done as @juckter did it)

Only use $Vieta$ relations... It's easy to a problem $3)$...

By Lagrange's Interpolation Formula, there exists a unique quadratic polynomial whose graph passes through $(r,s)$, $(s,t)$, $(t,r)$. (This already guarantees that the polynomial we will get works). However if this polynomial is $x^2+ax+b$, in the same way as in the above solutions (fixing typos), we get that the polynomial we want is $x^2+2x-14$. We are done. $\square$

There's an issue with your solution, which is that there is no reason for the Lagrange Interpolation polynomial to have leading coefficient $1$. You can expand this argument to get a pretty nice solution to the problem though.

You're right. Thank you for pointing out that mistake! Here is how I fixed it. (I wonder if there's a nicer way, though). Let $P_1(x)$ be the quadratic polynomial whose graph passes through $(r,s),(s,t),(t,r)$. Similarly, let $P_2(x)$ be the quadratic polynomial whose graph passes through $(r,t),(t,s),(s,r)$. By Lagrange's Interpolation Formula, they are both well-defined and unique. Thus, our problem reduces proving that one among $P_1(x),P_2(x)$ is monic. After that, we find $a,b$ as in the above solutions. Claim. $P_1(x)\ne P_2(x)$ Proof. Proceed by contradiction. If they are both equal to $P(x)=cx^2+ax+b$, then $P(x)$ passes through the six points mentioned above. In particular, it passes thorugh $(r,s)$, $(s,r)$, from which we get $r+s=-\frac{a+1}{c}$. By symmetry, this would imply $r+s=r+t=s+t$, a contradiction. Now, if $P_1(x)=cx^2+ax+b$, by doing the same computations and Vieta as in the above solutions, we get $(c,a,b)=\bigg(c,\dfrac{5c-1}{2},-14c\bigg)$. However, we also have $\sum_{cyc}(cr^2+ar+b)^2=\sum_{cyc}s^2$. Expanding, using Vieta, and simplifying stuff, this yields $c=\pm1$. However, if we do the same with $P_2(x)=c'x^2+a'x+b'$, we get $(c',a',b')=\bigg(c',\dfrac{5c'-1}{2},-14c'\bigg)$ and $c'=\pm1$. Finally, by the claim, this implies one of them has leading coefficient 1. We find it is $x^2+2x-14$, and are done. $\square$ Remark. With this method, we find that the other polynomial that permutates the roots cyclically is $-x^2-3x+14$.

Quote: Claim. $P_1(x)\ne P_2(x)$ Is it not enough for showing this to note that $P_1(x)$ and $P_2(x)$ take (by definition) different values for $x = r$, since the hypothesis states $s \not = t$?

baenanabread wrote: Quote: Claim. $P_1(x)\ne P_2(x)$ Is it not enough for showing this to note that $P_1(x)$ and $P_2(x)$ take (by definition) different values for $x = r$, since the hypothesis states $s \not = t$? True, that makes it easier Thank you!

Fun Fact. The roots are: $r=2\cos\frac{\pi}{63}+4\cos\frac{2\pi}{63}-2\cos\frac{4\pi}{63}-2\cos\frac{8\pi}{63}+2\cos\frac{10\pi}{63}-2\cos\frac{11\pi}{63}+4\cos\frac{16\pi}{63}-2\cos\frac{\pi}{9}\approx 3.13776$ $s=2\cos\frac{4\pi}{63}+2\cos\frac{11\pi}{63}+2\cos\frac{13\pi}{63}-2\cos\frac{17\pi}{63}-4\cos\frac{19\pi}{63}-2\cos\frac{22\pi}{63}+4\cos\frac{26\pi}{63}+2\cos\frac{4\pi}{9}\approx 2.12108$ $t=-4\cos\frac{5\pi}{63}-2\cos\frac{11\pi}{63}+2\cos\frac{17\pi}{63}-2\cos\frac{20\pi}{63}-4\cos\frac{23\pi}{63}+2\cos\frac{29\pi}{63}-2\cos\frac{31\pi}{63}+2\cos\frac{2\pi}{9}\approx -5.25885$