For convenience, consider the reciprocal polynomial $Q(z)=z^nP(1/z)=a_n+a_{n-1}z+\dots+a_0z^n$ and relabel the coefficients $b_0$ through $b_n$. We then wish to show that $|b_k|\leq 1-|b_0|^2$ for any $1\leq k \leq n$. If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant, so assume $|Q(z)|<1$ if $|z|<1$, and in particular $|b_0|<1$. If $\omega$ is a $k$th root of unity, then $Q(z)+Q(\omega z)+\dots+Q(\omega^{k-1}z)=k(b_0+b_kz^k+b_{2k}z^{2k}+\dots)$, so the polynomial $b_0+b_kz+b_{2k}z^2+\dots$ satisfies the same conditions. Thus, we may also assume $k=1$. Consider $\varphi(z)=\frac{z-b_0}{1-\overline{b_0}z}$. This is analytic with magnitude less than 1 in the open unit disk and sends $b_0$ to $0$. Therefore, $\varphi\circ Q$ has a power series $c_1z+c_2z^2+\dots$ which converges on the open unit disk and satisfies \[ |c_1|=\left|\frac{1}{2\pi i}\int_{|z|=r}\!\frac{\varphi(Q(z))}{z^2}\,\mathrm{d}z\right|\leq \frac{1}{2\pi}\int_{|z|=r}\!\left|\frac{\varphi(Q(z))}{z^2}\right|\,\mathrm{d}z\leq \frac{1}{2\pi}\int_{|z|=r}\!\frac{1}{|z|^2}\,\mathrm{d}z=\frac1r \]for all $0<r<1$, so $|c_1|\leq 1$. We also have that \[ |c_1|=|(\varphi \circ Q)'(0)|=|\varphi'(Q(0))|\cdot |Q'(0)|=\frac{1}{1-|b_0|^2}\cdot |b_1|, \]so $|b_1|\leq 1-|b_0|^2$ and we are done.

SwimWithDolphin wrote: For convenience, consider the reciprocal polynomial $Q(z)=z^nP(1/z)=a_n+a_{n-1}z+\dots+a_0z^n$ and relabel the coefficients $b_0$ through $b_n$. We then wish to show that $|b_k|\leq 1-|b_0|^2$ for any $1\leq k \leq n$. If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant, so assume $|Q(z)|<1$ if $|z|<1$, and in particular $|b_0|<1$. If $\omega$ is a $k$th root of unity, then $Q(z)+Q(\omega z)+\dots+Q(\omega^{k-1}z)=k(b_0+b_kz^k+b_{2k}z^{2k}+\dots)$, so the polynomial $b_0+b_kz+b_{2k}z^2+\dots$ satisfies the same conditions. Thus, we may also assume $k=1$. Consider $\varphi(z)=\frac{z-b_0}{1-\overline{b_0}z}$. This is analytic with magnitude less than 1 in the open unit disk and sends $b_0$ to $0$. Therefore, $\varphi\circ Q$ has a power series $c_1z+c_2z^2+\dots$ which converges on the open unit disk and satisfies \[ |c_1|=\left|\frac{1}{2\pi i}\int_{|z|=r}\!\frac{\varphi(Q(z))}{z^2}\,\mathrm{d}z\right|\leq \frac{1}{2\pi}\int_{|z|=r}\!\left|\frac{\varphi(Q(z))}{z^2}\right|\,\mathrm{d}z\leq \frac{1}{2\pi}\int_{|z|=r}\!\frac{1}{|z|^2}\,\mathrm{d}z=\frac1r \]for all $0<r<1$, so $|c_1|\leq 1$. We also have that \[ |c_1|=|(\varphi \circ Q)'(0)|=|\varphi'(Q(0))|\cdot |Q'(0)|=\frac{1}{1-|b_0|^2}\cdot |b_1|, \]so $|b_1|\leq 1-|b_0|^2$ and we are done. Why did you write: If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant?

Math-Problem-Solving wrote: SwimWithDolphin wrote: For convenience, consider the reciprocal polynomial $Q(z)=z^nP(1/z)=a_n+a_{n-1}z+\dots+a_0z^n$ and relabel the coefficients $b_0$ through $b_n$. We then wish to show that $|b_k|\leq 1-|b_0|^2$ for any $1\leq k \leq n$. If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant, so assume $|Q(z)|<1$ if $|z|<1$, and in particular $|b_0|<1$. If $\omega$ is a $k$th root of unity, then $Q(z)+Q(\omega z)+\dots+Q(\omega^{k-1}z)=k(b_0+b_kz^k+b_{2k}z^{2k}+\dots)$, so the polynomial $b_0+b_kz+b_{2k}z^2+\dots$ satisfies the same conditions. Thus, we may also assume $k=1$. Consider $\varphi(z)=\frac{z-b_0}{1-\overline{b_0}z}$. This is analytic with magnitude less than 1 in the open unit disk and sends $b_0$ to $0$. Therefore, $\varphi\circ Q$ has a power series $c_1z+c_2z^2+\dots$ which converges on the open unit disk and satisfies \[ |c_1|=\left|\frac{1}{2\pi i}\int_{|z|=r}\!\frac{\varphi(Q(z))}{z^2}\,\mathrm{d}z\right|\leq \frac{1}{2\pi}\int_{|z|=r}\!\left|\frac{\varphi(Q(z))}{z^2}\right|\,\mathrm{d}z\leq \frac{1}{2\pi}\int_{|z|=r}\!\frac{1}{|z|^2}\,\mathrm{d}z=\frac1r \]for all $0<r<1$, so $|c_1|\leq 1$. We also have that \[ |c_1|=|(\varphi \circ Q)'(0)|=|\varphi'(Q(0))|\cdot |Q'(0)|=\frac{1}{1-|b_0|^2}\cdot |b_1|, \]so $|b_1|\leq 1-|b_0|^2$ and we are done. Why did you write: If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant? Maximum modulus principle

@Ethan Is there any official solution??

SwimWithDolphin wrote: For convenience, consider the reciprocal polynomial $Q(z)=z^nP(1/z)=a_n+a_{n-1}z+\dots+a_0z^n$ and relabel the coefficients $b_0$ through $b_n$. We then wish to show that $|b_k|\leq 1-|b_0|^2$ for any $1\leq k \leq n$. If $|Q(z)|=1$ at some point in the open unit disk then $Q$ must be constant, so assume $|Q(z)|<1$ if $|z|<1$, and in particular $|b_0|<1$. If $\omega$ is a $k$th root of unity, then $Q(z)+Q(\omega z)+\dots+Q(\omega^{k-1}z)=k(b_0+b_kz^k+b_{2k}z^{2k}+\dots)$, so the polynomial $b_0+b_kz+b_{2k}z^2+\dots$ satisfies the same conditions. Thus, we may also assume $k=1$. Consider $\varphi(z)=\frac{z-b_0}{1-\overline{b_0}z}$. This is analytic with magnitude less than 1 in the open unit disk and sends $b_0$ to $0$. Therefore, $\varphi\circ Q$ has a power series $c_1z+c_2z^2+\dots$ which converges on the open unit disk and satisfies \[ |c_1|=\left|\frac{1}{2\pi i}\int_{|z|=r}\!\frac{\varphi(Q(z))}{z^2}\,\mathrm{d}z\right|\leq \frac{1}{2\pi}\int_{|z|=r}\!\left|\frac{\varphi(Q(z))}{z^2}\right|\,\mathrm{d}z\leq \frac{1}{2\pi}\int_{|z|=r}\!\frac{1}{|z|^2}\,\mathrm{d}z=\frac1r \]for all $0<r<1$, so $|c_1|\leq 1$. We also have that \[ |c_1|=|(\varphi \circ Q)'(0)|=|\varphi'(Q(0))|\cdot |Q'(0)|=\frac{1}{1-|b_0|^2}\cdot |b_1|, \]so $|b_1|\leq 1-|b_0|^2$ and we are done. The inequality $|b_1|+|b_0|^2$ follows from the Schwarz–Pick theorem which says that \[\dfrac{|f(z)'|}{1-|f(z)|^2}\leq \dfrac{1}{1-|z|^2}\]for any analytic function $f$ from the open unit disk to the open unit disk. See https://en.wikipedia.org/wiki/Schwarz_lemma

699th post! Might not be so hard if familiar with Parseval's theorem. Use $z\zeta$ to replace $z,$ and change $a_n$ to $|a_n|,$ WLOG $a_t,a_n\in\mathbb R_+.$ Now we prove the following Lemma: Quote: (Parseval's theorem) Polynomial $Q(z)=\sum_{k=0}^mc_kz^k,$ then $$\sum_{k=0}^n|c_k|^2=\frac 1{2\pi}\int_0^{2\pi}|Q(e^{i\theta})|^2d\theta.$$ Proof.$$\frac 1{2\pi}\int_0^{2\pi}|Q(e^{i\theta})|^2d\theta=\frac 1{2\pi}\int_0^{2\pi}\sum_{k,j=0}^nc_k\overline{c_j}e^{(k-j)i\theta}d\theta=\sum_{k,j=0}^nc_k\overline{c_j}\frac 1{2\pi}\int_0^{2\pi}e^{(k-j)i\theta}d\theta=\sum_{k=0}^n|c_k|^2.\Box$$Now back to original problem, \begin{align*}|a_n|^2+1&=\frac 1{2\pi}\int_0^{2\pi}|a_n+e^{(n-t)i\theta}|^2d\theta\\&\ge\frac 1{2\pi}\int_0^{2\pi}|P(e^{i\theta})(a_n+e^{(n-t)i\theta})|^2d\theta\\&=\frac 1{2\pi}\int_0^{2\pi}\left|\sum_{k=0}^{n-t-1}a_na_ke^{ik\theta}+\sum_{k=n-t}^{n}(a_na_k+a_{t+k-n})e^{ik\theta}+\sum_{k=n+1}^{2n-t}a_{k+t-n}e^{ik\theta}\right|^2d\theta\\&=\sum_{k=0}^{n-t-1}|a_na_k|^2+\sum_{k=n-t}^{n}|a_na_k+a_{t+k-n}|^2+\sum_{k=n+1}^{2n-t}|a_{k+t-n}|^2\\&\ge |a_n^2+a_t|^2+|a_n|^2.\end{align*}Since $a_n,a_t$ are positive real we get $a_n^2+a_t\le 1.\Box$