Hint

Functional_equation wrote: $-1\leq f(1)\leq 1$ Maybe it means $0\leq f(1)\leq1$?

@above Yes, that's right. $P(1,1)\implies 0\leq f(1)\leq 1$

Answer

@above I think (0,0) doesn't give the third solution

I looked at 5 cases to solve the problem. It took a long time.I made a mistake in one of the cases. I corrected it.

$P(0,0)$ yields $(f(0)-1)\cdot f(0)\le0$. Since $f(0)$ is integer, this implies $f(0)\in\{0,1\}$. (Case 1): $f(0)=0$. Then $P(x,0)$ yields $f(x)^2\le0$. Hence $f(x)\equiv0$ in this case. (Case 2): $f(0)=1$. (Case 2a): $f(x)\equiv1$ satisfies the conditions. (Case2b): There exists $a\in\mathbb{R}$ with $f(a)=0$. Consider an arbitary $x\in\mathbb{R}$ with $x\ne0$. Then $P(a/x,x)$ yields $f(a/x)^2+f(x)^2\le0$, and hence $f(x)=0$. This means that in this case $f(x)\equiv 0$ for all $x\ne0$ and $f(0)=1$.

Clearly, if $x=y=0$, we get $f(0)^2\leq f(0)$, implying $f(0)=0,1$. First, assume $f(x)$ has a zero at $r$. Then, \[f(x)^2+f\left(\frac rx\right)^2\leq 2f(r)=0\]implying $f(x)\equiv 0$ for $x\neq 0$ and $f(0)\in\{0,1\}$. Otherwise, we get that \[f(0)^2+f(x)^2\leq 2f(0)\implies f(x)^2\leq 1\]Clearly it is absurd if $f(r)=-1$ as then \[f(x)^2+f\left(\frac rx\right)^2\leq 2f(r)=-2\]Thus, $f(x)\equiv 1$.

Another way to phrase the inequality step is to say that the set \[ B = \{x\in \mathbb R: f(x) = 1\} \]is an ideal in $\mathbb R$. That is, if $b\in B$ and $a\in \mathbb R$ are arbitrary, then \[ 2f(ab) \geq f(a)^2 + f(b)^2 = 1, \]so $f(ab) \geq 1$ (it's an integer!) and hence $ab\in B$. But the only ideals of $\mathbb R$ are $\{0\}$ and $\mathbb R$ (and technically $\varnothing$, I guess); these correspond to the three possible solutions.