Is this correct? . Suppose , that $x=c$ is a integer solution to this equation . Then, $2abc^4-a^2c^2-b^2-1=0 \implies ac^2(2bc^2-a)=b^2+1$ . Note that $b^2+1$ prime factors are either $2$ or of form $4k+1$. Also we see that if , $a$ is even then , we get $4|b^2+1$ a contradiction . similarly we see that $c$ is also odd. And indeed if $b$ is odd , then we again get a similar contradiction. Now , if $a \equiv 3 \pmod{4}$ then , we it means there is prime factor of $a$ which is of form $4k+3$ a condtradiction. Also $c$ must also be of form $4k+1$ . And $b$ is even . But then we have $2bc^2-a^2 \equiv 3 \pmod{4}$ a contradiction .

Yep, it's correct. Another solution: The equation can be rewritten as $(ax+b)^2+1=2abx(x^3+1)$, which is impossible in $\mathbb{Z}/4\mathbb{Z}$ as far as one between $x$ and $x^3+1$ has to be even.

Rewrite the equation $b^2-2ax^4b+a^2x^2+1=0$ assuming it has integral solutions. Viewing the equation for $b$, since the discriminant should be a square, we have $a^2x^8 -a^2x^2-1=m^2$ for some integer $m$. Taking $\pmod 7$, we have $-1 \equiv m^2 \pmod 7$ which is an absurdity.