$|a-1|\geq\frac{\sqrt{3}}{4}$ or $a\in(-\infty,1-(\frac{\sqrt{3}}{4})]\cup[(\frac{\sqrt{3}}{4}+1),\infty)$ maybe.

i had got $a\in[\frac{2-\sqrt(2)}{2}]\cup[(\frac{2+\sqrt(2)}{2}]$

Calculation mistake

The answer is 1-(1/√2) < a < 1+(1/√2) And a is not equal to 0.5 and 1.5

Hint: Express $b$ in terms of $a$ using GP condition and as $P$ has real roots, $a^2\geq b$.

Here is my solution. An Important bound: Since the roots of $P(x)=0$ are positive reals, it follows that $D \ge 0 \implies 4a^2 \ge 4b \iff \boxed{a^2 \ge b}$. Now we exploit the $\text{GP}$ condition. Since $P(0), P(1)$ and $P(2)$ are in a geometric progression, let: $\rightarrow P(0) =b; P(1) = b+1-2a = b\cdot r ; b+4-4a = b\cdot r^2$. Thus, $b\cdot (b+4-4a) = (b+1-2a)^2$ $\implies b^2 + 4b - 4ab = b^2 + 1 + 4a^2 + 2b -4a - 4ab \iff 2b = (2a-1)^2 \implies |2a-1| = \sqrt{2b}$. Case 1: $a > \dfrac{1}{2} \implies 2a-1 = \sqrt{2b}$. Therefore, $a = \dfrac{\sqrt{2b}+1}{2}$. Now, our bound gives: $\dfrac{\sqrt{2b}+1}{2} = a \ge b^2 \iff 2b + 2\sqrt{2b} + 1 \ge 4b \iff 2b - 2\sqrt{2b} - 1 \le 0$. Now, we treat this as a quadratic in $\sqrt{2b}$. Hence, let $\sqrt{2b} = x$. Thus, $x^2-2x-1 \le 0$. From wavy-curve method, it is easy to obtain that $x \in [ 1-\sqrt{2} , 1 + \sqrt{2}] \implies \sqrt{2b} \in [0 , 1 \sqrt{2}]$ (as $\sqrt{2b} \ge 0$. Hence, $ a = \dfrac{\sqrt{2b}+1}{2} \in (\dfrac{1}{2} , \dfrac{ 2+ \sqrt{2}}{2})$. Case 2: $a < \dfrac{1}{2} \implies (1-2a) = \sqrt{2b}$. Therefore, $a = \dfrac{1- \sqrt{2b}}{2}$. Using the bound we get, $(\dfrac{1- \sqrt{2b}}{2})^2 \ge b \iff 1 + 2b -2\sqrt{2b} \ge 4b \iff 1-2\sqrt{2b} \ge 2b$. Similar to the previous case, set $\sqrt{2b} = y$. Hence, we obtain $y^2 + 2y - 1 \le 0$. Wavy-curve gives: $\rightarrow x \in [-\sqrt{2}-1 , \sqrt{2}-1] \implies \sqrt{2b} \in [0 , \sqrt{2}-1] \implies a = \dfrac{1-\sqrt{2b}}{2} \in [\dfrac{2-\sqrt{2}}{2} , \dfrac{1}{2})$. Now, taking the union of the two cases,we have : $\boxed{a \in [\dfrac{2-\sqrt{2}}{2} , \dfrac{1}{2}) \cup (\dfrac{1}{2} , \dfrac{3}{2}) \cup (\dfrac{3}{2} , \dfrac{ 2+ \sqrt{2}}{2}]}$.

A simpler solution I guess : Let x1 and X2 be roots of P(x). Then : [1-(1-x1)^2][1-(1-x2)^2]=(1-x1)^2 (1-x2)^2. This , (1-x1)^2 + (1-x2)^2=1. So let: 1-x1=sin(t) and 1-x2 =cos(t). a = 1-[sin(t)+cos(t)]/2=1- [sin(t+45)]/√2. So (1-1/√2)<a<(1+1/√2)

As $P(x)$ has real roots, we use $D \geq 0$ to get $a^2\geq b$. Now using the condition that $P(1)^2=P(0).P(2)$, we get $(2a-1)^2=2b \implies b=\frac{(2a-1)^2}{2}$ Now putting the value of $b$ in our inequality, we get $2a^2-4a+1 \leq 0$ Now using the wavy curve method, we get $a \in \left[1-\frac{1}{\sqrt2}\right] \cup \left[1+\frac{1}{\sqrt2}\right]$ Also, keeping in mind that $P(0).P(1).P(2)\neq 0$, we have that $a \neq \frac{1}{2}$, $\frac{3}{2}$.