$f(P)=\pm P$? After getting $f(f(P))=P$, bounding degree of $f(P)$ between that of $P$ $/pm 1$, forcing degree of $f(P)$ equal to P by considering odd/even degree P, getting $f(Q)=kQ$ for certain types of Q (aka ones with distinct roots), hence forcing $f(P)=kP$ for all $P$ (variable k) by taking degree of Q to infinity along with having $P-Q, f(P)-f(Q)$ having equal roots, forcing $k$ to be fixed for all polynomials and finally forcing $k= \pm 1$ should work?

I will post the official solution (with minor tweaks):

This was definitely the hardest problem, although I am not sure..

{redacted}

This is brutal rip

Exactly @above orz

anantmudgal09 wrote: Let $\mathbb{R}[x]$ be the set of all polynomials with real coefficients. Find all functions $f: \mathbb{R}[x] \rightarrow \mathbb{R}[x]$ satisfying the following conditions: $f$ maps the zero polynomial to itself, for any non-zero polynomial $P \in \mathbb{R}[x]$, $\text{deg} \, f(P) \le 1+ \text{deg} \, P$, and for any two polynomials $P, Q \in \mathbb{R}[x]$, the polynomials $P-f(Q)$ and $Q-f(P)$ have the same set of real roots. Proposed by Anant Mudgal, Sutanay Bhattacharya, Pulkit Sinha Amazing problem , here's my solution from test.

Also I feel P5 was hardest problem on test .

Pretty neat question! I have a suspicion that $\deg f(P) \le 1 + \deg P$ is not entirely necessary. I did use this in my official write-up but not in a crucial way. Even the condition $f(0) = 0$ can probably be relaxed, if one includes the solutions $P(x) \equiv - x + c$ in the solution set. I'll try to finish that later, for now a solution that assumes everything the problem gives. Call a polynomial sweet if it has only distinct real roots. Given any polynomials $Q_1$ and $Q_2$, we can pick a polynomial $P$ such that all three of $P, P-Q_1, P-Q_2$ are sweet. Observe for any sweet $P$, $f(P)$ has same real roots as $P$, so $P \mid f(P)$ The idea is to say that if both $Q_1, Q_2$ are bounded by $(-d, d)$ on $[0,1]$, we can choose $P$ of large degree $2k$ by $P(\frac{i}{2k}) = -1^i \dot d$ for $0 \le i \le 2k$ using Lagrange Interpolation. Now, each of the polynomials $P, P-Q_1, P-Q_2$ has a root in the intervals $(\frac{i}{2k}, \frac{i+1}{2k})$ by intermediate value theorem, and hence is sweet. Pick arbitrary $Q_1, Q_2$ and $P$ by above strategy. Let $f(P) = aP$ $P-f(Q_k) \mid f(P) - Q_k$ and by matching the leading two coefficients, $a(P - f(Q_k)) = f(P) - Q_k$ and $a f(Q_k) = Q_k$. Since $Q_1, Q_2$ were arbitrary we have that $\frac{Q}{f(Q_1)}$ is a constant. It is not hard to analyze to find out that this constant must be $\pm 1$.

What's with the title Oh noice, thanks@above

Very nice problem! My solution is same as Bob's upto claim 7 so I will not post that but I think the following finish is cleaner. Case 1: $a=1$, now $P-c$ and $f(P)-c$ have the same set of real roots. But now this set of real roots must also be a root of $P-f(P)$ but as we can vary $c$ and generate arbitrary roots of $P-c$ thus, $P-f(P)=0\implies P=f(P)$. Similarly, Case 2: $a=-1$, now $P-c$ and $f(P)+c$ have same set of real roots and thus these are also roots of $P+f(P)$ and again by same logic, we have $f(P)=-P$ and both these are valid solutions and we are done!

Consider polynomial $P$ $\deg 0$. If $f(P)$ is a polynomial of degree at most $1$ with no real roots, so a constant also. If $P$ has a repeated real root $c$ then $f(P)$ could be the polynomial with $ (x-c)f(P)=P.$ Let $ f(P)=Q$ If $f$ is an involution on the constants ( $f(Q)=P$) . For the constants $Q$ $\forall P$ we have $f(P)-f(Q)$ has the same real roots as $P-Q$. Let $P$ be linear polynomial. Now $f(P )$ is at most quadratic which has the same unique real root as $ P$ so of the form $ cP$ or $ cP^2$ with $c$ nonzero. For any constant $ Q$ we have $f(P )-f(Q)$ is one of $c(P-Q ), c(P-Q)^2.$ Note that $f(P)-f (Q)$ had the same degree as $f(P)$. Suppose that $f(ax+b)=c(ax+b)^2$ Then for a nonzero constant $Q $ of the opposite sign of $c,f(ax+b)-Q$ had no real roots but $ax+b-f(Q)$ had. Note that, $f(P)=c(P)P$ for each linear polynomial $P$. $f$ is an involution of degree 1 also. Consider $f(ax+b)=c(ax+b)$. $ax+b-Q$ has the same root as $c(ax+b)-f(Q)\,\,\forall Q \implies f(Q)=cQ$. Note that, $c$ is only depended on $ax+b$. There is a single constant $c$ such that $cP=f(P)\,\forall P$ of degree 1 at most. Note that $f$ is an involution on the constants $\implies c^2=1$ either $f(P)=-P$ or $P= f(P)$ for these polynomials. Let $P $ be a polynomial degree $n, $ and suppose $P $ has the largest possible number of distinct real roots in the coset $P\mathbb{R}[x]^{\le n-2} $ of its translated by the polynomials of degree at most $n-2. $ If $P $ has less than $n $ real roots then either has repeated real roots or an irreducible quadratic factor . We have $P=RS $ where $R $ has degree $n-2$ and $S $ is quadratic with at most one real root. Choose $a $ such that $S-a $ has $2$ real roots of $R $. Then $P-Ra $ has more real roots than P, contradiction. Note that, $P $ has $n $ real roots. For any real polynomial $P $ of degree $n $ there is a polynomial $Q $ of degree at most $n-2$ such that $P-Q $ has $n $ real roots. Lemma- Let $c $ be one of $\pm 1, $ and by induction forall the polynomials $Q $ of degree $< n $ we have $cQ=f (Q). $ Let $P $ have degree $n. $ Suppose $f (P) $ has degree $n+1. $ By the lemma there is $Q $ of degree at most $n-1$ such that $f (P)-Q $ has $n+1$ distinct roots. Then $f (Q)-P $ requires $n+1$ roots, not possible. We conclude that $f (P) $ has degree $n. $

Let $P(P,Q)$ correspond to condition 3. $P(f(Q),Q): f(f(Q))=Q \implies f$ is bijective. $P(P,f(Q)): P-Q, f(P)-f(Q)$ have the same set of real roots. Plugging in $Q=0$, $P, f(P)$ have the same set of real roots. As $f(f(Q))=Q, \deg Q \leq \deg f(Q)+1 \implies \deg f(Q)+1 \geq \deg f(Q) \geq \deg Q -1$. If $\deg P \neq \deg f(P)$, this forces one of $P, f(P)$ to be of even degree and the other to be of odd degree. If $\deg f(P)$ is even, we can choose $Q$ as a constant polynomial such that $f(P)-Q$ has no real roots, whereas $P-f(Q)$ must have at least one real root as it has an odd degree $\implies$ Contradiction. Hence we have $\deg f(P) =\deg P$ for $\deg P \geq 3$. Let $Q(x)=A(x-a_1)...(x-a_n)$ for distinct $a_1,a_2,...,a_n$. As $Q, f(Q)$ have the same set of real roots $\implies f(Q)=kQ$. Note that, as $P-Q$ and $f(P)-f(Q)=f(P)-kQ$ have the same set of real roots (say $x_1,x_2,...,x_t$), $x_1,x_2,...,x_t$ must also be roots of $f(P)-kP$. Clearly we can let degree of $Q$ be as large as we want it to be and between any 2 roots of Q we can let it's local maxima/minima as large in magnitude as we can, $f(P)-kP$ must be the zero polynomial $\implies f(P)=kP$ for some variable real $k$. Consider two polynomials $Q_1,Q_2$ with pairwise distinct roots, as above. Consider $P$ with pairwise distinct real roots also such that the set of roots of $P$ is a superset of $Q_1,Q_2$. Let $P(x)=Q_1(x)R_1(x) \implies (R_1-1)Q, k(R_1-\frac c{k_1})Q$ must have the same set of real roots. Similarly, $(R_2-1)Q_2, (R_2-\frac c{k_2})Q_2$ must have the same set of real roots. As we can let $\deg R$ to be as large as we want along with the fact that $t$ is a root of $R_1-1$ and $R_1- \frac{c}{k_1}$, then $t$ is a root of $1-\frac{c}{k_1}$, we hence force $c=k_1=k_2$. This forces all $k$ equal in the above argument for $f(P)=kP$. As $f(f(P))=P \implies k^2=1 \implies k = \pm 1$. Thus $f(P)=P$ and $f(P)=-P$ are the only possible solutions, both which clearly work.

Can somebody check plz? After proving f(c)=c or f(c)=-c for all real c, WLOG f(c)=c for all real c, consider Q=P(c) for some real c, and f(P)=R(x) we have P-P(c) has root c, then R(x)-P(c) also has root c or R(c)=P(c), and since it happen for all such real c we must have R(x)=P(x) or f(P)=P

Sorry if there is any typos or error, 7 Solved with TLP.39 & EpicNumberTheory. The first assertion in the problem is just $f(0) = 0$. Let the second assertion be denoted by $A(P)$ and the third assertion by $B(P,Q)$. $B(f(Q),Q) \implies 0 \text{ and } Q - f(f(Q))$ have the same set of real roots. So $f(f(Q)) = Q$ and so $f$ is an involution. $B(0,Q) \implies f(Q) \text{ and } Q$ have the same real roots (!) Suppose $\text{deg } R = 1$. So $A(R) \implies \text{deg } f(R) \in \{0,1,2\}$. Now by (!) we have that $f(R) \in \{cR,cR^2,C\}$ (Here $C$ is any constant polynomial). I Claim 1. $ \text{deg } R \neq 2$ Proof. So for proving that $f(R)$ can’t be $cR^2$, we need $cR^2-f(d)$ and $R-d$ to not have the same set of real root for some $d$. Simply choose $d$ such that $f(d)$ have the different sign from $c$ and $cR^2-f(d)$ will not have any real root and thus a contradiction (this is possible as $f$ is involution.) $\square$. I Claim 2. $\text{deg } R \neq 0$ Proof. Suppose $f(R_1)=C$ for some $R_1$.$A(C) \implies \text{deg } f(C) \in \{0,1\}$. Suppose it is $1$ then by $B(C,0) $ we have that $f(C)$ and $0$ have the same set of real roots or that $f(C) = 0 \implies \text{deg } f(C) = 0 \neq 1$ which is a clear contradiction. $B(R_1,C) \implies R_1-f(C) \text{ and } 0$ have the same set of real roots. So $R_1= f(C)$ which means that $\text{deg }R_1 = 0$ which is a clear contradiction. Let $R,R’$ have different roots .Consider $B(R,R’)$ which gives that $cR-R’\text{ and } R-dR’$ have the same set of roots, which means that there’s a constant $m$ such that $m(cR-R’)=R-dR’$ and since $R,R’$ have different roots we must have $mc=1$ and $-m=-d$. This gives $cd=1$.. Simply choose $R_2$ with different root from both of them and let $f(R_2)=kR_2$ , then we have $cd=dk=ck=1$ and so $c=d=\pm 1$. So any $R,R’$ with different roots have the same $\frac{f(R)}{R}$ which is $1$ or $-1$. Let $\text{deg }T = 1$. Suppose $f(T) = T$ but $f(T')=-T'$. Then $B(T,T')$ gives that $T+T'$ and $T'-T$ have the same set of real roots which contradicts their degree being equal to $1$. So we have either $f(R)=R$ $\forall R \in \mathbb{R}[x]$ or $f(R) = -R$ $\forall R \in \mathbb{R}[x]$. L Case 1. $f(R) = R$ $\forall R \in \mathbb{R}[x]$. $B(R,Q) \implies f(Q) - R \text{ and } Q - R$ have the same set of real roots. So now varying $R$ over $\mathbb{R}[x]$ we can deduce that$$\boxed{f(Q) = Q \text{ } \forall Q \in \mathbb{R}[x]}$$which indeed is a solution. L Case 2. $f(R) = -R$ $\forall R \in \mathbb{R}[x]$. $B(R,Q) \implies f(Q) - R \text{ and } Q + R$ have the same set of real roots. So now varying $R$ over $\mathbb{R}[x]$ we can deduce that$$\boxed{f(Q) = -Q$ $\forall Q \in \mathbb{R}[x]}$$which also is a solution.

$Q=f(P)$ shows that $f(f(P))=P$. $Q=0$ shows that $P$ and $f(P)$ have the same set of roots. If $P(x)=c \ne 0$ is constant, $f(P)$ can be at most linear, but has no roots, so in fact $f(P)=g(c)$ is also constant and $g(g(c))=c$. Putting $Q=g(c)$ shows that $P(y)=c$ iff $f(P)(y)=g(c)$, in other words $\boxed{f(P)(y)=g(P(y))}$ for all $y \in \mathbb{R}$. Specifically, for $P_0(x)=x$ this shows that $f(P_0)(y)=g(y)$ and hence $f(P_0)$ is bijective. On the other hand, $f(P_0)$ is at most quadratic, hence linear (since bijective) and hence $f(P_0)(x)=ax$ (since $0$ is a root). Hence $g(y)=ay$ and hence $a=\pm 1$. If $a=1$, then $f(P)=P$ which is indeed a solution. If $a=-1$, then $f(P)=-P$ which is indeed a solution.

Anyone has idea regarding cutoff ? 20 maybe ?

Can someone check this solution? My friend submitted this on the test. Please do point out the mistakes, if any.

Umm, there's no reason that $f$ is a polynomial. We are just given that $f$ is a function.

I'll borrow the $\sim$ notation from anantmudgal09 above Plugging in $Q = f(P)$ gives $f(f(P)) = P$, and plugging in $Q= 0$ gives $P\sim f(P)$. Call a polynomial good if all it's roots are real and distinct. Using the Fundamental theorem of Algebra, and the fact $deg f(P) \leq deg P +1$, we deduce $P\sim f(P) \implies f(P)=P\cdot const$ for a good polynomial $P$. Now take good polynomials $P$ and $P(x-a)$, and say $f(P)=Pc$ and $f(P(x-a))=P(x-a)d$. Now we have $P-P(x-a)d \sim P(x-a)-Pc \implies \frac{1}{d}=c$. Using simple induction on the degree of $P$, we get that the constant in $f(P)=P\cdot const$ is the same for all $P$ of the same degree, where the constant is $c$ or $1/c$ depending on the parity of the degree of $P$. We can easily transform $P-f(Q)\sim Q-f(P)$ into $P-Q\sim f(P)-f(Q)$ due to involutivity. Now use good polynomials $P$ and $Px$ here to get $Px-P\sim Px\frac{1}{c}-Pc \implies c\in\{-1,1\}$. So $f(P)=P$ or $f(P)=-P$ for any good $P$. Here I'll skip a huge chunk of the proof and just quote an equivalent of a past USAMO problem: For any $Q\in \mathbb{R}[x]$, there are two good polynomials $P, R$ with $P-Q=R$. The original problem does need some tweaking though, but it is pretty simple. Take any real polynomial $Q$, and find such $P$ and $R$, also make their degrees larger then that of $Q$. Then $P-Q\sim f(P)-f(Q) \implies \pm R \sim \pm P - f(Q)$. From here we easily conclude $f(Q)=\pm Q$, finishing the proof. Thus, the only solutions are $f(P) = P$ and $f(P) = -P$ $ \blacksquare$

BOBTHEGR8 wrote: anantmudgal09 wrote: Let $\mathbb{R}[x]$ be the set of all polynomials with real coefficients. Find all functions $f: \mathbb{R}[x] \rightarrow \mathbb{R}[x]$ satisfying the following conditions: $f$ maps the zero polynomial to itself, for any non-zero polynomial $P \in \mathbb{R}[x]$, $\text{deg} \, f(P) \le 1+ \text{deg} \, P$, and for any two polynomials $P, Q \in \mathbb{R}[x]$, the polynomials $P-f(Q)$ and $Q-f(P)$ have the same set of real roots. Proposed by Anant Mudgal, Sutanay Bhattacharya, Pulkit Sinha Amazing problem , here's my solution from test.

Also I feel P5 was hardest problem on test . If you find P5 tough, go see AMC 12B, 2008 P24. You will start cursing me or yourself...

@above I think their is a typo somewhere, I saw AMC 12B ,2008 P24, it has nothing to do with P5 .

Here is my solution, video link: https://youtu.be/wwYb0fqrWMQ Plug $Q=0$ in (iii) yields $P$ and $f(P)$ has the same set of real roots, call this property ($\star$). Claim 1: $f$ sends constant polynomials to constant polynomials Indeed take $P=c$ for $c \neq 0$, then $deg(f(c)) \le 1$. From ($\star$), $f(c)$ has no real root, note that any degree 1 polynomial has a real root, hence $f(c)$ can only be degree $0$, i.e. constant polynomials. Claim 2: $f(X) = \alpha X$ for some $\alpha \neq 0$. (Here $X$ refers to the polynomial $P(x)=x$) From $(\star)$ and the degree condition, $f(X)$ can only be $\alpha X$ or $\beta X^2$. However, if $f(X) = \beta X^2$, then take $P=X$ and $Q = \beta$ gives (1) $P-f(Q) = X-f(\beta)$ is degree 1, which has 1 root; and (2) $Q-f(P) = \beta - \beta X^2 $ has two root $\pm 1$, contradiction! Claim 3: $f(c) = c/\alpha$ for any $c \in \mathbb{R}$. Apply (iii) with $P=X$ and $Q=c$ gives $P-f(Q) = X -f(c)$ has the same root as $Q-f(P) = c - \alpha X$. Comparing the root gives $f(c) = c/\alpha$. Claim 4: $f(P) = \alpha P$ for any polynomial $P$. Fix $P$, take an arbitrary real number $x_0 \in \mathbb{R}$. Let $Q = \alpha P(x_0)$, then $f(Q) = P(x_0)$. So $P-f(Q) = P- P(x_0)$ has the same real real root set as $Q-f(P) = \alpha P(x_0) - f(P)$. As $x_0$ is a solution of $P-P(x_0)$, this means $x_0$ is also a solution of $\alpha P(x_0) - f(P)$. Hence $f(P)(x_0) = \alpha P(x_0)$. This is true for any $x_0$, hence $f(P) = \alpha P$. Finally, from Claim 3 and Claim 4, $f(1) = 1/\alpha = \alpha$, so $\alpha^2 =1$. Hence the only two solutions are $f(P)=P$ for all $P$, and, $f(P)=-P$ for all $P$.

Consider the function , $f~:~R[X] \rightarrow R[X]$ , which satisfies : (a) $f(0)~=~0$ , (b) for any non-zero polynomials $P~\in~R[X]~,~degf(P)~\le~1+degP$ , and (c)for any two polynomials $P,Q \in R[X]$ , the polynomials $f(P)-Q$ and $f(Q)-P$ have the same set of real zeroes. Observe , that on considering $Q(x)=f(P(x))$ , we get that , $0=f(P(x))-f(P(x))$ and $f(f(P(x)))-P(x)$ have the same set of real zeroes.Hence , $f(f(P(x)))=P(x)$ , (otherwise we get a contradiction on the maximality of zeroes of $f(f(P(x)))=P(x)$).Also , we have $f(P(x))=0$ $<=>$ $P(x)=0$ , where "$<=$" follows from condition(a) , and "$=>$" follows from $f(P(x))=0$ $=>$ $f(f(P(x)))=f(0)=0$ $=>$ $P(x)=0$.Again , observe that $f(P(x))$ and $P(x)$ have that same set of real zeroes. Claim:$f(P(x))=P(x)$ and $f(P(x))=-P(x)$ are only such functions which satisfy the above conditions . $\Rightarrow$ $deg f(P(x)) \le 1+deg P(x)$ $\Rightarrow$ $deg f(f(P(x))) \le 1+deg f(P(x))$ $\Rightarrow$ $deg P(x) \le 1+ deg f(P(x))$ , hence we are left with with symmetric inequalities : $deg f(P(x)) \le 1+deg P(x)$ and $deg P(x) \le 1+ deg f(P(x))$ , and thus we get , $deg P(x)-1 \le deg f(P(x)) \le deg P(x)+1$ $\Rightarrow$ $deg f(P(x)) \in \{deg P(x)-1 , deg P(x) , deg P(x)+1\}$. We now show that $f$ maps constant polynomials to constant polynomials:$f(0)=0$ , take $k \neq 0$ , thus $deg f(k) \le 1$ .Lets consider $deg f(k)=1$ , then its a linear polynomial which has exactly one real zero and hence we see that $k$ and $f(k)$ doesn't have same set of real roots , which is a contradiction , hence $deg f(k)=0$. Now if , $deg f(P(x))=deg (P(x)) \pm 1$ , then if $deg f(P(x))$ is even , then obviously $deg (P(x))$ has to be odd , and $f(P(x))$ has finite local minima's below $x$-axis and $\lim_{x \to \infty} f(P(x))$ $\rightarrow$ $\infty$ , now consider $Q(x)=-max\{f(P(x_1)) , f(P(x_2)) , \cdots , f(P(x_l)) \}+1$ which is a constant polynomial where $x_i's ~,~ 1 \le i \le l$ are the points of local minima's of $f(P(x))$ , hence $f(P(x))-Q(x)$ has no real zeroes , but $P-f(Q(x))$ is of odd degree and should have at least one real zero , hence a contradiction.A similar type of argument follows for the case where $deg f(P(x))$ is odd.Therefore , $deg f(P(x))=deg P(x)$ . Will complete the rest of the proof sometime later.

Notation: $P\sim Q$ if P and Q have the same real roots. $Q=0 \implies P\sim f(P)$. If $P=a \neq 0 \implies deg(f(P)) \leq 1$. But $deg(f(P)) \neq 1$, because $P$ does not have roots $\implies deg(f(P))=0$. If $P=ax+b \implies deg(f(P))\leq 2$, but $P\sim f(P) \implies f(ax+b)=K_{a,b}(ax+b)$ or $f(ax+b)=K_{a,b}(ax+b)^2$, $K_{a,b}\neq 0$. $(ax+b)-f(c)\sim c-f(ax+b) \implies$ 1) $(ax+b)-f(c)\sim c-K_{a,b}(ax+b)^2 \implies c=K_{a,b}f(c)^2 \implies$ $c$, $K_{a,b}$ have the same signal, but you can put $c>0$ or $c<0$. Absurd! 2) $(ax+b)-f(c)\sim c-K_{a,b}(ax+b) \implies c=K_{a,b}f(c)$, varying $a, b \implies K_{a, b}$ is constant, so, $f(ax+b)=K(ax+b)$ and $f(c)=\dfrac{c}{K}$. $ax+b-K(cx+d) \sim cx+d-K(ax+b) \implies \dfrac{b-Kd}{a-Kc}=\dfrac{d-Kb}{c-Ka} \implies K^2(ad-bc)=(ad-bc)$ (just put $ad \neq bc$) $\implies K^2=1 \implies K=\pm 1$. 1) $K=1 \implies f(a)=a \implies P-a \sim f(P)-a$, just take $a=P(r) \implies f(P)(r)=P(r)$ for an infinite number of $r$'s $\implies f(P)=P$. 2) $K=-1 \implies f(-a)=a \implies P-a \sim -f(P)-a$, analogously, $f(P)=-P$.

anantmudgal09 wrote: Let $\mathbb{R}[x]$ be the set of all polynomials with real coefficients. Find all functions $f: \mathbb{R}[x] \rightarrow \mathbb{R}[x]$ satisfying the following conditions: $f$ maps the zero polynomial to itself, for any non-zero polynomial $P \in \mathbb{R}[x]$, $\text{deg} \, f(P) \le 1+ \text{deg} \, P$, and for any two polynomials $P, Q \in \mathbb{R}[x]$, the polynomials $P-f(Q)$ and $Q-f(P)$ have the same set of real roots. Proposed by Anant Mudgal, Sutanay Bhattacharya, Pulkit Sinha Nice and easy problem. Let \(\omega(P)\) denote the assertion of the second bullet point, and \(\Omega(P,Q)\) of the third. From \(\Omega(f(P),P)\), it is clear that \(f\) is an involution. Now, \(\Omega(P,0)\) tells us that \(f(P)\) and \(P\) have the same set of real roots, implying that \(f(x)=cx^k\), where \(c\) is a real constant, but \(\omega(x)\) tells us that the degree of \(f(x)\) is at most \(2\), implying that \(f(x)=cx\) or \(f(x)=cx^2\). If \(f(x)=cx^2\), then \(\Omega(x,C)\), where \(C\) is a constant gives us that \(C-cx^2\) and \(x-f(C)\) have the same set of real roots, and since \(f(C)\) is either constant or linear by \(\omega(C)\), we get a contradiction since \(C-cx^2\) has 2 real roots, while \(x-f(C)\) has just one. Therefore, \(f(x)=cx\). Now, I claim that \(f(C)=\frac{C}{c}\) for all constants \(C\). By \(\Omega(C,x)\) and \(\omega(C)\), we see that \(f(C)\) is either linear or constant and that \(C-cx\) and \(x-P(C)\) have the same real roots. If \(f(C)\) was linear, then it'd not be unique because we could let \(f(C)-x=k(C-cx)\) to get infinitely many values for \(f(C)\), a contradiction, therefore \(f(C)\) is constant, and checking gives \(f(C)=\frac{C}{c}\). Now, note that \[C=f(f(C))=\frac{C}{c^2}\]implying that \(c=\pm1\). If \(c=1\), then \(f(x)=x\) and \(f(C)=C\), and \(\Omega(P,C)\) gives us that \(P-C\) and \(f(P)-C\) have the same set of real roots, implying that \(P-C\) and \(f(P)-P\) have the same set of real roots. Fixing \(P\) and varying \(C\), we see that \(f(P)=P\) for all polynomials \(P\in\mathbb{R}[x]\). Analogously, if \(c=-1\), then \(f(P)=-P\) for all polynomials \(P\in\mathbb{R}[x]\). In conclusion, the only solutions to this functional relation is \[f(P)=P~\text{or}~f(P)=-P\forall P\in\mathbb{R}[x]\]

The answer is $f(P)=P$ and $f(P)=-P$, which clearly work. We show that these are the only ones. First, by plugging in $Q=f(P)$, it follows that $f(f(P))=P$, so we can rewrite the condition as $P-Q$ and $f(P)-f(Q)$ having the same set of real roots. In particular, $P$ and $f(P)$ have the same set of real roots. Now, I claim that the second condition can be strengthened to $\deg f(P) \leq \deg P$. First, since $P$ and $f(P)$ have the same set of real roots, if $P$ is a nonzero constant then $f(P)$ must be too, since it is at most linear and every linear polynomial has a real root, i.e. $f$ maps nonzero constant polynomials to nonzero constant polynomials. Then, if $\deg f(P)>0$ is even, there exists some choice of constant $c$ such that $P-c$ has no real roots, so $f(P)-f(c)$ should have no real roots, i.e. its degree should be even too, hence $\deg f(P) \neq 1+\deg P$ which is odd. Then due to the fact that $f$ is an involution, if $\deg P$ is odd then $\deg f(P)$ cannot be even, else $\deg f(f(P))$ is even, so $\deg P \neq 1+\deg P$. Therefore, if $P$ is some nonconstant polynomial with all roots real and distinct, then $f(P)$ must be some nonzero multiple of $P$, i.e. it equals $aP$ for some $a \neq 0$. For some sufficiently small nonzero $\varepsilon$, $P-\varepsilon$ has all roots real and distinct as well, and $aP-f(\varepsilon)$ has the same roots. Since their degrees are the same, it follows that they are multiples of each other, i.e. $f(\varepsilon)=a\varepsilon$. On the other hand, since $f$ is an involution, if $\varepsilon$ is small enough then we can apply the same argument to $aP-\varepsilon$ and $P-f(\varepsilon)$ to get that $f(\varepsilon)=\tfrac{1}{a}\varepsilon$, hence $a=\pm 1$, which implies that $f(P)=\pm P$. Furthermore, it is clear that this argument implies that the choice of sign must be identical for every polynomial with all roots real and distinct, since given two such polynomials $P_1$ and $P_2$ we may choose some $\varepsilon$ small enough for both and apply the argument on both polynomials. Finally, given an arbitrary polynomial $P$, let $Q$ be a polynomial with $\deg Q>\deg P$ whose roots are all real and distinct. Then for sufficiently large real $M$, the roots of $MQ-P$ are also all real and distinct, and the same as the roots of $f(MQ)-f(P)$. If $f(MQ)=MQ$ then the polynomials $MQ-P$ and $MQ-f(P)$ have the same leading coefficient, degree, and set of $\deg Q$ roots, hence they are the same, so $f(P)=P$. Otherwise, if $f(MQ)=-MQ$ then the polynomials $MQ-P$ and $-MQ-f(P)$ have opposite leading coefficient, and the same degree and set of $\deg Q$ roots, hence they are opposites of each other, so $f(P)=-P$. This implies that $f(P)=P$ for all $P$, or $f(P)=-P$ for all $P$, as desired. $\blacksquare$

Claim.$f(f(P)) = P$ for every $P \in \mathbb{R}[x]$. Proof. Just put $P = P, Q = f(P)$ in the third assertion. This tells us that $P-f(f(P))$ and $0$ have the same set of real roots. Thus, $P - f(f(P)) = 0 \implies P = f(f(P))$. $\blacksquare$ Claim. $f(c) = \pm c$ for every constant polynomial $c$. Proof. First we show that $f(c)$ is a constant polynomial for every constant polynomial $c$. Put $P = c, Q = 0$ in the third assertion. This says that $c$ and $f(c)$ have the same set of real roots, i.e. no roots. However, by the second assertion, $f(c)$ is linear or constant. If it were linear, it would have a real root, impossible. Thus, $f(c)$ is a constant. Now we show that, in fact, $f(c) = \pm c$. Suppose $a$ was a constant such that $f(a) = b \neq a$. WLOG assume that $b > a$ (otherwise swap $a$ and $f(a)$ and use $f(f(a)) = a$). Case 1. $a>0$. Let $P = x^2, Q = b$ in the third assertion. This means that $x^2-a$ and $f(x^2)-b$ have the same set of real roots. Clearly $f(x^2)$ is not a constant or linear, because then $f(x^2)-b$ would not have two real roots. Suppose $f(x^2)$ was cubic. Then $f(x^2)-b$ would have a double root at either $-\sqrt{a}$ or $\sqrt{a}$. WLOG assumes that the double root is at $\sqrt{a}$. Let $P = x^2, Q = a$ in the third assertion. This means that $x^2-b$ and $f(x^2)-a$ have the same set of real roots. The real roots of $x^2-b$ are $\pm \sqrt{b}$, but $f(x^2)-a$ either has only one real root, or a root in $(\sqrt{a}, \sqrt{a})$, both impossible. Thus, $f(a) = a$ for any constant $a$. Case 2. $a<0$. Now put $P = f(x^2+2a), Q = b$. If $f(x^2+2a)$ was cubic, an almost identical analysis as case 1 gives a contradiction. So suppose that $f(x^2+2a)$ was a quadratic polynomial. Then $P=x^2+2a, Q = b$ gives that $f(x^2+2a) = c(x^2+a)+b$, and $P = x^2 + 2a, Q = a$ gives that $f(x^2+2a) = c(x^2+2a-b) + a$. Equating them gives $c = -1$. and putting $P = x^2+2a, Q = b$ and noting that $f(x^2+2a) = -x^2-2a$ gives that $-x^2-2a - b = -x^2-a \implies a = -b$. Now we show that we cannot have $f(a) = -a, f(b) = b$ for two distinct non-zero constants $a, b$. Suppose this were the case. Then let $P = x^2 + a, Q = a$ first, giving $f(x^2+a) + a = tx^2$ and then let $P = x^2 + a, Q = b$ giving $f(x^2 + a)-b = s(x^2 + a - b)$. Equating them gives $s = t = \frac{b+a}{b-a}$. So $f(x^2 + a) = \frac{b+a}{b-a}x^2 - a$. LHS is independent of $b$, but RHS is dependent on $b$. Put $b = 0$ giving $f(x^2 + a) = -x^2-a$ and then put non-zero $b$ giving $f(x^2 + a) = \frac{b+a}{b-a}x^2 - a$, however $\frac{b+a}{b-a} \neq -1$ for non-zero $a, b$. This is a contradiction, thus we cannot have both $f(a) = -a$ and $f(b) = b$ for distinct non-zero $a, b$. $\blacksquare$ To finish, we consider the two possible cases. Case 1. $f(c) = c$ for every constant $c$. observe that by letting $P = P, Q = c$ in the third assertion, we get that $P-c$ and $f(P) - c$ have the same set of real roots for \textit{any} real number $c$. Thus if $\alpha = P(t) \neq f(P)(t) = \beta$, then $P(x)-alpha$ has $t$ as a root, but not $f(P)(x) - \alpha$, contradiction. Thus, $P(x) = f(P)(x)$ for every $x$, thus $P = f(P)$. Case 2. $f(c) = -c$ for every constant $c$. Putting $P = P, Q = c$ gives $P+c$ and $f(P)-c$ have the same set of real roots for every constant $c$. This is impossible unless $f(P) = -P$. Thus, either $f(P) = P$ for every $P$ or $f(P) = -P$ for every $P$. $\square$