By Cauchy-Schwarz Inequality, $(\frac{1}{2}a^2+b^2+\frac{1}{2}a^2)(2x^6+x^4+2x^2)\geq (ax^3+bx^2+ax)^2=(x^4+1)^2$ So $a^2+b^2\geq \frac{(x^4+1)^2}{2x^6+x^4+2x^2}\geq\frac{4}{5}$ The last inequality is equivalent to $5(x^8+2x^4+1)\geq 4(2x^6+x^4+2x^2) 0\Leftrightarrow (x^2-1)^2(5x^4+2x^2+5)\geq 0$

It is also necessary to prove that if $a^2+b^2\ge\frac45$ then the given equation has at least one real root.

Since $x=0$ isn't the solution of the given equation, we can rewrite as follows. $x^{4}+ax^{3}+bx^{2}+ax+1=0\Longleftrightarrow x^{2}+ax+b+\frac{a}{x}+\frac{1}{x^{2}}=0\ (x\neq 0).$ By $\left|x+\frac{1}{x}\right|=|x|+\frac{1}{|x|}\geq 2\sqrt{|x|\cdot\frac{1}{|x|}}=2$ when the eqality holds when $|x|=\frac{1}{|x|}\ (x\neq 0)\Longleftrightarrow x=\pm 1.$ Let $x+\frac{1}{x}=t,$ we are to find the condition for which the equation with respect to $t, \ t^{2}+at+b-2=0$ has at least one real root at $t\leq-2\ or\ t\geq 2.$ The condition for which $a,\ b$ should satisfy this is given by $2|a|-2\leq b\leq \frac{1}{4}a^{2}+2$ and $|a|\geq 4\ \cdots [A],\ |b+2|\geq2|a|\ \cdots [B].$ Sketching the distribution satisfying $[A]$ or $[B]$ on $a-b$ plane gives the answer, $a^{2}+b^{2}$ has the minimum value of $\frac{4}{5}$ when $(a,\ b)=\left(\pm \frac{4}{5},\ \frac{2}{5}\right)$ which are the intersection points between two graphs of $b=2|a|-2$ and $b=-|a|.$

Lemma. For $k>0$ the function $f: [k,\infty)\rightarrow R\ ,\ f(x)=x+\frac{k^{2}}{x}$ is increasing. Particularly, $0<k\le a\le x\Longrightarrow a+\frac{k^{2}}{a}\le x+\frac{k^{2}}{x}$ with equality iff $x=a\ .$ Indeed, for $0\le k\le u<v$ we have $f(u)-f(v)=\frac{(u-v)(uv-k^{2})}{uv}<0$, i.e. $f(u)<f(v)$ because $uv>0$, $u-v<0$ and $u\ge k>0\ ,\ v>k>0\Longrightarrow uv>k^{2}\ .$ Example. $m=\min_{x\ne 0}\frac{\left(x^{4}+1\right)^{2}}{x^{2}\left(x^{4}+3x^{2}+1\right)}=$ $\min_{y>0}\frac{\left(y^{2}+1\right)^{2}}{y\left(y^{2}+3y+1\right)}=$ $\min_{y>0}\frac{\left(y+\frac{1}{y}\right)^{2}}{\left(y+\frac{1}{y}\right)+3}=$ $\min_{z\ge 2}\frac{z^{2}}{z+3}=$ $\min_{t\ge 5}\frac{(t-3)^{2}}{t}$, where $x\ne 0$ $\Longleftrightarrow$ $y=x^{2}>0$ $\Longleftrightarrow$ $z=y+\frac{1}{y}\ge 2$ $\Longleftrightarrow$ $t=z+3\ge 5\ .$ Thus, $m=\min_{t\ge 5}\left(t+\frac{9}{t}-6\right)=-6+\min_{t\ge 5}\left(t+\frac{9}{t}\right)\ .$ Using the above lemma $(3=k<a=5\le t)$, obtain $m=-6+\left(5+\frac{9}{5}\right)=\frac{4}{5}$ and the minimum points are $x\in \{\pm 1\}$ because $t=5\Longleftrightarrow z=2\Longleftrightarrow y=1\Longleftrightarrow x\in \{\pm\ 1\}\ .$ The proof of the proposed problem. Can consider that the relation $x(x^{2}+1)\cdot \underline a+x^{2}\cdot \underline b+(x^{4}+1)=0$, $x\in R^{*}$ is the equation of a mobile line $d$in the analytical plane $aOb\ .$ Thus, square of the distance $PO$, where $P(a,b)\in d\ ,\ OP\perp d$ is $a^{2}+b^{2}=\frac{(x^{4}+1)^{2}}{x^{2}(x^{2}+1)^{2}+(x^{2})^{2}}=\frac{(x^{4}+1)^{2}}{x^{2}(x^{4}+3x^{2}+1)}\equiv f(x)\ .$ Therefore, $m=\min_{P(a,b)\in d}\{a^{2}+b^{2}\}=\min_{x\ne 0}f(x)=\frac{4}{5}$ and $x\in \{\pm 1\}$ (from the above example), i.e. the minimum points are the solutions of the systems $2\epsilon a+b+2=0\ ,\ a-2\epsilon b=0$, where $\epsilon^{2}=1\ .$ In conclusion, $a_{\mathrm{min}}=\frac{4\epsilon}{5}$ and $b_{\mathrm{min}}=-\frac{2}{5}$, where $\epsilon^{2}=1\ .$

Proof for the proposed problem. Since $x=0$ isn't the solution of the given equation, we can rewrite as follows. $x^{4}+ax^{3}+bx^{2}+ax+1=0\Longleftrightarrow x^{2}+ax+b+\frac{a}{x}+\frac{1}{x^{2}}=0\ (x\neq 0).$ By $\left|x+\frac{1}{x}\right|=|x|+\frac{1}{|x|}\geq 2\sqrt{|x|\cdot\frac{1}{|x|}}=2$ when the eqality holds when $|x|=\frac{1}{|x|}\ (x\neq 0)\Longleftrightarrow x=\pm 1.$ Let $x+\frac{1}{x}=t,$ then $|t|\ge 2$ and the given equation becomes: $t^{2}+at-b-2=0\iff t^{2}-2=-at-b$ We have: $(a^{2}+b^{2})(t^{2}+1)\ge (at+b)^{2}=(t^{-}2)^{2}\implies a^{2}+b^{2}\ge\frac{t^{4}-4t^{2}+4}{t^{2}+1}\quad (1)$ On the otherhand, $$\frac{t^{4}-4t^{2}+4}{t^{2}+1}-\frac{4}{5}=\frac{(5t^{2}-4)(t^{2}-4)}{5(t^{2}+1)}\ge 0\quad (2)$$ From $(1)$ and $(2)$ we have: $a^{2}+b^{2}\ge\frac{4}{5}$

Remark. We also have: $a^{2}+(b-2)^{2}>3$ Indeed, we have: $$t^{2}=-(at+b-2)\implies t^{4}=\left(at+b-2\right)^{2}\le\left[a^{2}+(b-2)^{2}\right]\left(t^{2}+1\right)\\ \implies a^{2}+(b-2)^{2}\ge\frac{t^{4}}{t^{2}+1}>t^{2}-1>3$$ And if the given equation is: $x^{4}+ax^{3}+bx^{2}+cx+1=0$ then $a^{2}+b^{2}+c^{2}\ge\frac{4}{3}$

a nice solution pco but i am asking too about [the equation of a mobile line]i need more informations because it s the first time time that i heard about that.

After making the substitution t=x+1/x, analyse the 3 different cases. Either both roots of the quadratic are greater than 2 in absolute value and have same sign or both are greater than 2 in absolute value and have opposite sign or there is exactly 1 root between -2 and 2. Using properties of quadratic expressions we get the desired result.