Since, $P(0)^2=b^2 \in \mathbb{Z} \implies b \in \mathbb{Z} $, now, $$(a+b+1)^2 \in \mathbb{Z} \implies a^2+2a(b+1) \in \mathbb{Z} $$$$(2a+b+4)^2 \in \mathbb{Z} \implies 4a^2+4a(b+4) \in \mathbb{Z} $$$$4a^2+4a(b+1)+12a-2(a^2+2a(b+1)) \in \mathbb{Z} \implies 4a(a+3) \in \mathbb{Z} $$So, we can only have $a=\frac{c}{2} \not \in \mathbb{Z} $, which contradicts the fact that $P(1)^2 \in \mathbb{Z} $. Hence, $a,b \in \mathbb{Z} $

khan.academy wrote: Since, $P(0)^2=b^2 \in \mathbb{Z} \implies b \in \mathbb{Z} $, now, $$(a+b+1)^2 \in \mathbb{Z} \implies a^2+2a(b+1) \in \mathbb{Z} $$$$(2a+b+4)^2 \in \mathbb{Z} \implies 4a^2+4a(b+4) \in \mathbb{Z} $$$$4a^2+4a(b+1)+12a-2(a^2+2a(b+1)) \in \mathbb{Z} \implies 4a(a+3) \in \mathbb{Z} $$So, we can only have $a=\frac{c}{2} \not \in \mathbb{Z} $, which contradicts the fact that $P(1)^2 \in \mathbb{Z} $. Hence, $a,b \in \mathbb{Z} $ I think this solution is wrong as 4a(a+3) is an integer, does not imply a = c/2 for some integer c. (As a side note, check the official solution for the correct solution to this problem)