A little hard

An interesting question which came to my mind was if it can be solved for $f(0)=0$ and the answer is positive, with the only solutions are $f(x)=x$ for every $x\in\mathbb{R}$ or $f(x)=0$ for every $x\in\mathbb{R}$ and if you want to ask me if I was proposer of this the answer is no.

Let $P\left(x,y\right)$ the equation of the text. Putting $P\left(0,0\right)$ we have $$-f\left(0\right)^2=\max\left\{2f\left(0\right),f\left(0\right)\right\}\stackrel{f\left(0\right)\ne0}{\Rightarrow}\max\left\{2f\left(0\right),f\left(0\right)\right\}<0\Rightarrow f\left(0\right)<0\Rightarrow$$$$\Rightarrow\max\left\{2f\left(0\right),f\left(0\right)\right\}=f\left(0\right)\Rightarrow-f\left(0\right)^2=f\left(0\right)\stackrel{f\left(0\right)\ne0}{\Rightarrow}f\left(0\right)=-1$$Putting $P\left(x,0\right)$ we have $$f\left(x\right)^2=-2f\left(x\right)+\max\left\{f\left(x^2\right)-1,f\left(x^2\right)\right\}=-2f\left(x\right)+f\left(x^2\right)\Rightarrow f\left(x^2\right)=f\left(x\right)\left(f\left(x\right)+2\right)$$From the last equation it also follows that $f\left(-x\right)=f\left(x\right)\vee-2-f\left(x\right)$ Define $A=\left\{x:\ f\left(x\right)=f\left(-x\right),\ x\in\mathbb{R}\right\},\ B=\left\{x:\ f\left(-x\right)=-f\left(x\right)-2,\ x\in\mathbb{R}\right\}$. Of course $A\cup B=\mathbb{R}$ and iff $x\in A\cap B$ then $f\left(x\right)=-1$. Now we'll show that $A=\mathbb{R}\vee B=\mathbb{R}$. Suppose for the sake of contraddiction that there exists a real $a\in A\setminus B$ (hence $a\ne0$) and a real $b\in B\setminus A$. Comparing $P\left(x+a,a\right)$ and $P\left(x+a,-a\right)$ we obtain that $f\left(x+2a\right)=\pm f\left(x\right)$, hence $f\left(2a\right)=\pm1$. Also note that if $k\in A$ then $f\left(k\right)\ne 1$, because by $P\left(k,-k\right)$ we get $1=2+\max\left\{2f\left(k^2\right),f\left(2k^2\right)\right\}\le 2+2f\left(k^2\right)=2+2f\left(k\right)\left(f\left(k\right)+2\right)=8$ which is absurd. Hence $f\left(a\right)\ne -1$ because $a\not\in A\cap B$ and $f\left(a\right)\ne 1$ because $a\in A$. We will also prove that $f\left(a\right)\ne 0$. If it doesn't happen there are two cases to be analysed: $\text{Case}\ 1:\ f\left(a\right)=0,\ f\left(2a\right)=-1$ We have $f\left(4a^2\right)=f\left(2a\right)\left(f\left(2a\right)+2\right)=-1$ and $f\left(a^2\right)=f\left(a\right)\left(f\left(a\right)+2\right)=0$. Comparing $P\left(x,0\right)$ and $P\left(x,2a\right)$ we have that $$f\left(x^2\right)=\max\left\{f\left(x^2\right)-1,f\left(x^2-4a^2\right)\right\}\Rightarrow f\left(x^2\right)=f\left(x^2+4a^2\right)$$and it follows that $f$ is periodic with period $4a^2$, so $f\left(2a^2\right)=f\left(-2a^2\right)\Rightarrow 2a^2\in A$. Now look at $P\left(a,a\right)$ and obtain that $$1=\max\left(0,f\left(2a^2\right)\right)$$which let us say $f\left(2a^2\right)=1$, absurd because $2a^2\in A$. $\text{Case}\ 2:\ f\left(a\right)=0,\ f\left(2a\right)=1$ In this case note that $f\left(-2a\right)=\pm f\left(0\right)=\mp f\left(2a\right)$, hence if $f\left(-2a\right)=f\left(2a\right)$ it would mean $2a\in A$ (absurd because we are supposing $f\left(2a\right)=1$), if not $f\left(-2a\right)=-f\left(2a\right)=-1$ so we would have $2a\not\in B$ and $2a\not\in A$, absurd because $A\cup B=\mathbb{R}$. Comparing $P\left(a,b\right)$ and $P\left(-a,-b\right)$ we obtain $$f\left(a+b\right)^2-2f\left(a\right)f\left(b\right)=f\left(-a-b\right)^2-2f\left(a\right)f\left(-b\right)$$If $\left(a+b\right)\in A$ then $f\left(a\right)f\left(b\right)=f\left(a\right)f\left(-b\right)\stackrel{f\left(a\right)\ne0}{\Rightarrow}f\left(b\right)=f\left(-b\right)$ which can't happen because of how we defined $b$. Hence $\left(a+b\right)\in B$ (this allows us to say that forall $x\in B$ we have $\left(x+a\right)\in B$), so substituting $f\left(-a-b\right)=-f\left(a+b\right)-2$ and $f\left(-b\right)=-2-f\left(b\right)$ we get $$f\left(a+b\right)^2-2f\left(a\right)f\left(b\right)=\left(f\left(a+b\right)+2\right)^2+2f\left(a\right)\left(f\left(b\right)+2\right)\Rightarrow$$$$\Rightarrow f\left(+b\right)=-f\left(a\right)f\left(b\right)-f\left(a\right)-1$$so substituting $b\rightarrow b+a$ (we can do this because we already showed $\left(b+a\right)\in B\setminus A$) in the last one we get $$\pm f\left(b\right)=f\left(b+2a\right)=-f\left(a\right)f\left(b+a\right)-f\left(a\right)-1=$$$$=-f\left(a\right)\left(-f\left(a\right)f\left(b\right)-f\left(a\right)-1\right)-f\left(a\right)-1\stackrel{f\left(a\right)\ne \pm1}{\Rightarrow}f\left(b\right)=\frac{-f\left(a\right)^2+1}{f\left(a\right)^2\pm1}$$Since $f\left(b\right)\ne1$ (by $b\not\in A\cap B$) it must be asserted that $f\left(b\right)=\frac{-f\left(a\right)^2+1}{f\left(a\right)^2+1}$. But now, since of course also $-b\in B\setminus A$ we can put $b\rightarrow-b$ and say $$f\left(-b\right)=\frac{-f\left(a\right)^2+1}{f\left(a\right)^2+1}=f\left(b\right)$$contraddiction because $b\not\in A$ for hypothesis. Now suppose $A=\mathbb{R}$. Comparing $P\left(\frac{y+x}{2},\frac{y-x}{2}\right)$ and $P\left(\frac{y+x}{2},\frac{x-y}{2}\right)$ we get that $f\left(y\right)=\pm f\left(x\right)$, so $f\left(x\right)=\pm c,$ and since $f\left(0\right)=-1$ we get $f\left(x\right)=\pm1$. Hence looking at $P\left(\sqrt{\frac{x}{2}},\sqrt{\frac{x}{2}}\right)$ with $x\ge0$ we get $$-1=\max\left\{2f\left(\frac{x}{2}\right),f\left(x\right)\right\}$$and since $2f\left(\frac{x}{2}\right)\ne -1$ we get $$-1=f\left(x\right)\ \ \ \forall\ x\ge0$$Finally putting $P\left(x,y\right)$ with $x\ge 0$ and $y<0$ we get $-1=2f\left(y\right)-1\Rightarrow f\left(y\right)=-1\ \ \ \forall\ y<0$, hence the only solution with $A=\mathbb{R}$ is $f\left(x\right)=-1\ \ \ \forall x.$ At this point we are almost done, it remains the case with $B=\mathbb{R}$. Comparing $P\left(x,y\right)$ and $P\left(-x,-y\right)$ it follows that $$f\left(x+y\right)^2-2f\left(x\right)f\left(y\right)=f\left(-x-y\right)^2-2f\left(-x\right)f\left(-y\right)=$$$$=\left(f\left(x+y\right)+2\right)^2-2\left(f\left(x\right)+2\right)\left(f\left(y\right)+2\right)\Rightarrow f\left(x+y\right)=f\left(x\right)+f\left(y\right)+1$$Now if we define $g\left(x\right)=f\left(x\right)+1$, from last equality we have $g\left(x+y\right)=g\left(x\right)+g\left(y\right)$, and from $f\left(x^2\right)=f\left(x\right)\left(f\left(x\right)+2\right)$ we get $g\left(x^2\right)=g\left(x\right)^2$, hence $g$ satisfies Cauchy equation and the additional hypothesis $g\left(x\right)\ge 0\ \ \forall\ x\ge 0$, hence $g\left(x\right)=cx\Rightarrow f\left(x\right)=cx-1$. Substituting in the text we get $$\left(cx+cy-1\right)^2=2\left(cx-1\right)\left(cy-1\right)+\max\left\{cx^2+cy^2-2,cx^2+cy^2-1\right\}=$$$$=2\left(cx-1\right)\left(cy-1\right)+cx^2+cy^2-1$$hence from this we can easily say that the only solutions are $f\left(x\right)=x-1\ \ \forall\ x$ and $f\left(x\right)=-1\ \ \forall\ x.$

mathwizard888 wrote: Find all functions $f:\mathbb{R}\rightarrow\mathbb{R}$ such that $f(0)\neq 0$ and for all $x,y\in\mathbb{R}$, \[ f(x+y)^2 = 2f(x)f(y) + \max \left\{ f(x^2+y^2), f(x^2)+f(y^2) \right\}. \] (Solution with Shubham Saha) Answer: $x \mapsto -1$ and $x \mapsto x-1$ are the only such functions. It is easy to see that they work. Let $P(x, y)$ denote this functional equation. Note that $$P(0,0) \implies -f(0)^2=\operatorname{max} \{f(0), 2f(0) \} \implies f(0)=-1$$since $f(0) \ne 0$. Also, $$P(x, 0) \implies f(x^2)= f(x)^2+2f(x)$$for all $x$. Put $x \mapsto -x$ in the previous relation, to conclude that $$f(-x) \in \{f(x), -2-f(x) \}$$for all $x \in \mathbb{R}$. Case 1. $f$ is an even function. Compare $P(x, y)$ and $P(x, -y)$ to conclude that $f(x+y)^2=f(x-y)^2$ for all $x, y$; hence $|f|$ is a constant function. As $-1 \in \text{Im}(f)$ we get $f(x) \in \{-1, +1\}$ for all $x$, so if $f$ is nonconstant then pick $x, y$ with $f(x) \ne f(y)$ to get $$P(x, y) \implies 3=|\operatorname{max} \{f(x^2)+f(y^2), f(x^2+y^2)\}| \le 2,$$contradiction! Hence, $f \equiv -1$ in this case. $\blacksquare$ Case 2. Let $f \not \equiv -1$ now. Claim: $f(x)+f(-x)=-2$ for all $x \in \mathbb{R}$. (Proof) Suppose $t \in \mathbb{R}$ such that $s=f(x_0)=f(-x_0)$; we shall see that $s=-1$ to conclude. Note that $$P(x, y) -P(-x, -y) \implies f\left(-(x+y)\right)-f(x+y)=f(x)f(y)-f(-x)f(-y).$$Call the last equation $(+)$, then put $y \mapsto y+z$ in $(+)$ and substitute $x=x_0, y=-x_0$ to see that $$f(-z)-f(z)=s\left(f(z-x_0)-f(x_0-z)\right) \overset{(+)}{=} s^2\left(f(-z)-f(z)\right).$$As $f$ is not an even function, we obtain $s^2=1$. If $s=1$ then $$P(x_0, -x_0) \implies 1=2s^2+\operatorname{max} \{2f(x_0^2), f(2x_0^2)\} \ge 2s^2+2(s+1)^2-2,$$which is false, hence the claim is valid. $\blacksquare$ Finally, put $f(-x)=-2-f(x)$ in $(+)$ to get $$\boxed{f(x+y)=f(x)+f(y)+1}.$$Hence, $g(x) := 1+f(x)$ is an additive function and $g(x^2)=g(x)^2 \ge 0$ is bounded on an interval; hence $g$ is linear and it easily follows that $g$ is the identity function, just as we desired. $\blacksquare$

$P(0,0): f(0) = -1.$ $P(x,0): (f(x)+1)^2 = f(x^2)+1$. Let $g = f+1$, then $g(0) = 0, g(x^2) = g(x)^2\geq 0$ and the equation becomes \[g(x+y)^2-2g(x+y) = 2g(x)g(y)-2g(x)-2g(y)+\max(g(x^2+y^2), g(x)^2+g(y)^2-1)......R(x,y)\]Note that $g(x)^2 = g(x^2) = g(-x)^2$. Let $E:=\{x\in\mathbb{R}\mid g(x)=g(-x)\}$. Then $g(x) = -g(-x)\neq 0\quad\forall x\not\in E$. For any $a,b\in E$, $R(-a,b)-R(a,-b)$ implies that $a-b\in E$. $R(a,b),R(a,-b)$ together shows that \[(g(a+b)-1)^2 = (g(a-b)-1)^2......(1)\]If there exists $x\neq 0, y$ such that $g(x) = 0, y \not\in E$ and W.L.O.G. suppose that $x>0, y<0$ (and so $g(y)<0$). Since $x\in E, y\not\in E$, $x+y$ cannot be in $E$ (or else $y = (x+y)-x$ should falls in $E$) and $R(x,y)-R(-x,-y)$ implies that $g(x+y) = g(y)$. By induction, $g(nx+y)=g(y)\quad\forall n\in\mathbb{N}$. Choose a sufficiently large $n$ such that $nx+y>0$, then $0\leq g(nx+y) = g(y) < 0$, a contradiction. Hence, $g$ is even or $g(x) = 0\Leftrightarrow x = 0$. Case 1. $g$ is even. By (1), $(g(x)-1)^2$ is constant (and hence is 1), and so $g(x) = 0,2$. If there exists $c$ such that $g(c) = 2$, then $g(\sqrt{|c|}) = \sqrt{2}$, a contradiction. Hence $g(x) = 0$ and so $f(x) = -1$ in this case. Case 2. $g(x) = 0\Leftrightarrow x = 0$. If there exists $t\neq 0$ in $E$, then by plugging in $(t,t)$ in (1), one can deduce that $g(2t) = 0(\textup{impossible}),2$. Since $2t\in E$, $g(4t) = 2$ too. $R(2t, 2t): 0 = 0+\max(g(8t^2), 7) \geq 7$, a contradiction. Hence, $E = \{0\}$, and so $f(x) = -f(-x)\quad\forall x\in\mathbb{R}$. $R(x,y)-R(-x,-y)$ gives that $g(x+y) = g(x)+g(y)$, so it's well-known that $g(x) = 0$(impossible) or $g(x) = x$. Therefore, $f(x) = x-1$ in this case. In conclusion, $f(x) = -1$ or $f(x) = x-1$.

Here it is solution when $f(0)=0$.

Let $P(x,y)$ Denote the statement of the original equation. Then $P(0,0)$ gives $f(0)^2=2f(0)^2+\max\{2f(0),f(0)\}\Rightarrow 0\ge -f(0)^2=\max\{f(0),2f(0)\} \Rightarrow f(0)\le 0 \Rightarrow -f(0)^2=f(0) \Rightarrow f(0)=0, -1$. \underline{Case 1: $f(0)=0$} Then $P(x,0)$ gives $f(x)^2=f(x^2)$, so $f(x)\ge 0$ for all $x\ge 0$. Note that for $x,y> 0$, $$f(x+y)^2\ge 2f(x)f(y)+f(x^2)+f(y^2)\ge 2f(x)f(y)+f(x)^2+f(y)^2=[f(x)+f(y)]^2$$So $f(x+y)\ge f(x)+f(y)$, since both $f(x+y), f(x)+f(y)$ are non-negative for $x,y>0$. So $f(x^2+y^2)\ge f(x^2)+f(y^2)$ for any real $x,y$, so original equation becomes $$f(x+y)^2=2f(x)f(y)+f(x^2+y^2)$$Note that for $x,y>0$, $$2f(x)f(y)+f(x^2+y^2)=f(x+y)^2=f((x+y)^2)\ge f(x^2+y^2)+f(xy)+f(xy)$$So $f(xy)\le f(x)f(y)$ for all $x,y>0$. Now, $f(1)^2=f(1^2)=f(1) \Rightarrow f(1)=0, 1$. \underline{Case 1a: $f(1)=0$} Then $f(x)\le f(x)f(1)=0$ for all $x>0$, so together with $f(x)\ge 0$, we have $f(x)=0$ for all $x\ge 0$. But $f(x)^2=f(x^2)=f(-x)^2 \Rightarrow |f(-x)|=|f(x)|=0$ for all $x>0$. So $f(x)=0$ for all reals $x$. \underline{Case 1b: $f(1)=1$} Then $P(1,1)$ gives $f(2)^2=2f(1)^2+f(2)=2+f(2) \Rightarrow f(2)=2$, since $f(2)>0$. We will solve this like IMO 2013 Problem 5. First, $f(2^k)\le f(2)^k=2^k$, and for any $n\in\mathbb{N}$, $f(n)\ge nf(1)=n$. So $f(2^k)=2^k$. Now take any real $x$, then $$f(x)^k\ge f(x^k)\ge f(\lfloor x^k \rfloor)\ge \lfloor x^k \rfloor> x^k-1$$so $f(x)>\sqrt[k]{x^k-1}$. Take $k$ large then $f(x)\ge x$ for all reals $x$. For any $k\in \mathbb{N}$ and any positive real $x$, $x<2^k$, then $$2^k=f(2^k)\ge f(2^k-x)+f(x)\ge 2^k-x+x=2^k$$So equality holds, and $f(x)=x$ for all $x\in \mathbb{R}, x<2^k$. For any $x\in \mathbb{N}$, take $k$ with $x<2^k$, then $f(n)=n$ for all positive reals $x$. Finally, $f(x)^2=f(x^2)=f(-x)^2 \Rightarrow |f(x)|=|f(-x)|$. If $f(a)=f(-a)$ for some real $a$, then take the difference in $P(a,a), P(a,-a)$ gives $f(2a)^2-2f(a)f(a)=f(0)^2-2f(a)f(-a) \Rightarrow 4a^2=f(2a)^2=f(0)^2=0$, false. So $f$ is odd hence $f(x)=x$ for all reals $x$. \underline{Case 2: $f(0)=-1$} Then $P(x,0)$ gives $f(x)^2=-2f(x)+f(x^2)$, so $(f(x)+1)^2=f(x^2)+1=(f(-x)+1)^2 \Rightarrow f(x)=f(-x)$ or $f(x)+f(-x)=-2$ for all real $x$. Now define the two disjoint sets $$A=\{x:f(x)=f(-x)\}, B=\{x: f(x)+f(-x)=-2, f(x)\neq-1\}$$then every positive real $x$ belongs to exactly one of $A$ or $B$. We will restrain the structure of $A$ and $B$. First, if $f(c)^2=f(-c)^2$, then $c\in A$, or else $f(c)=-f(-c)$ and $c\in B$, but then $0=f(c)+f(-c)=2$, false. So $c\in A$. Now for every $a, b>0$, we make three observations: i) If $a\in A$, $b\in A$, take the difference in $P(a,b), P(-a,-b)$ gives $f(a+b)^2-2f(a)f(b)=f(-a-b)^2+2f(-a)f(-b)\Rightarrow f(a+b)^2=f(-a-b)^2$, so $a+b\in A$. ii) If $a\in A$, $b\in B$, if $a+b\in A$, take the difference in $P(a,b), P(-a,-b)$ gives $f(a+b)^2-2f(a)f(b)=f(-a-b)^2+2f(-a)f(-b) \Rightarrow f(a)=0$. So $f(a)=f(-a)=0$, take the difference in $P(a,y), P(-a,y)$ for every real $y$ gives $f(-a-y)^2-2f(-a)f(-y)=f(a+y)^2-2f(a)f(y)\Rightarrow f(a+y)^2=f(-a-y)^2\Rightarrow a+y\in A$ for every real $y$. Take $y=b-a$ yields $b\in A$, false. So $a+b\in B$. iii) If $a\in B$, $b\in B$, Then $f(-a)=-2-f(a)$ and $f(b)=-2-f(-b)$. Take the difference in $P(a,b), P(-a,-b)$ gives $f(a+b)^2-2f(a)f(b)=f(-a-b)^2+2f(-a)f(-b)$. So $$f(a+b)^2-f(-a-b)^2=2f(a)f(b)-2f(-a)f(-b)=2f(a)f(b)-2(-2-f(a))(-2-f(b))=-2(f(a)+f(b)+2)$$But recall that $f(a), f(b)\neq -1$ and $a,b>0 \Rightarrow f(a), f(b) \ge -1$, so $f(a), f(b)> -1$. So $f(a+b)^2-f(-a-b)^2\neq 0 \Rightarrow f(a+b)\neq f(-a-b)$. So $a+b\in B$. The three observations proves that $a+b\in A \iff a, b\in A$. Now we claim that $A$ or $B$ is null. Indeed, if exist some $a>0$ with $a\in A$, then $a+a\in A \Rightarrow 2a\in A$. Inductively we get $2^ka\in A$. For any positive real $x>0$, take $k$ with $0<x<2^ka$, then $x+(2^ka-x)=2^k\in A$, so $x\in A$ for every real $x>0$. If such $a$ does not exist then $x\in B$ for every $x>0$. This proves our claim. Now, $f(1)^2+2f(1)=f(1^2)=f(1)$, so $f(1)=0, -1$. \underline{Case 2a: $x\in B$ for all $x>0$} Note that $f(1)\neq -1$ otherwise $1\not\in B$, so $f(1)=0$. For every real $x, y>0$, take the difference in $P(x,y), P(-x,-y)$ gives $$f(x+y)^2-2f(x)f(y)=f(-x-y)^2+2f(-x)f(-y)=(-2-f(x+y))^2-2(-2-f(x))(-2-f(y))$$$$\Rightarrow f(x+y)=f(x)+f(y)+1$$Recall that $f(x^2)=(f(x)+1)^2-1$, so if we define $g(x)=f(x)+1$, then for all $x,y>0$, we have $g(x+y)=g(x)+g(y)$, $g(1)=1$, $g(x)\ge 0$ and $g(x)=-g(-x)$. Then $g(p)=pg(1)=p$ for all $p\in \mathbb{Q^+}$. Then for any positive real $x>0, p\in \mathbb{Q^+}$, $g(x+p)=g(x)+g(p)=g(x)+p$. So if $g(x)<x$, then choose $p\in \mathbb{Q^+}$ with $x>p>g(x)$, then $$p>g(x)=g(x-p)+g(r)>g(x-p)+r\ge p$$since $f(x-p)\ge 0$, false. Likewise if $g(x)>x$, then choose $p\in \mathbb{Q^+}$ with $x<p<g(x)$, then $$p<g(x)=g(x-p)+g(p)=g(x-p)+p\le p$$since $ g(x-p)=-g(p-x)\le 0$, again false. So $g(x)=x$ for all $x>0$, and together with $g$ is odd then $g(x)=x$ for all reals $x$. This means $f(x)=x-1$ for all reals $x$. \underline{Case 2b: $x\in A$ for all $x>0$} Since $f(x)=f(-x)$ for every real $x>0$, take the difference in $P(x,x), P(x,-x)$ gives $$f(2x)^2-2f(x)^2=f(0)^2+2f(x)f(-x) \Rightarrow f(2x)^2=f(0)^2=1$$so $f(x)=1, -1$ for every $x>0$. If $f(x)=1$ for some $x>0$, then $P(x,1)$ gives $$f(x+1)^2=-2f(x)+\max\{f(x^2)-1, f(x^2+1)\}=-2+\max\{f(x^2)-1, f(x^2+1)\}\le -2+1=-1$$since $|f(x^2)|, |f(x^2+1)|=1$. This is false, so $f(x)=1$ cannot hold. So $f(x)=-1$ for all $x>0$, and since $f$ is even, then $f(x)=-1$ for all reals $x$. Solution set and verfication: i) $f(x)=x$, gives $(x+y)^2=2xy+\max\{x^2+y^2,x^2+y^2\}=x^2+y^2+2xy$, true. ii) $f(x)=0$, gives $0^2=2(0)^2+\max\{2(0),0\}=0$, true. iii) $f(x)=x-1$, gives $(x+y-1)^2=2(x-1)(y-1)+\max\{(x^2-1)+(y^2-1),x^2+y^2-1\}=2xy-2x-2y+2+(x^2+y^2-1)=x^2+y^2+2xy-2x-2y+1$, true. iv) $f(x)=-1$, gives $(-1)^2=2(-1)^2+\max\{2(-1),-1\}=2+(-1)=-1$, true. So these are indeed solutions and are the only solutions. Q.E.D

CASE-1:if f[x^2+y^2] was maximum,then f[x+y]^2=2f[x]f[y]+f[x^2+y^2] if we put x=y=0,we get f[0]=2f[0]f[0]+f[0] =>f[0]=0,so this case is not possible CASE-2:if f[x^2]+f[y^2] is maximum f[x+y]^2=2f[x]f[y]+f[x^2]+f[y^2] if we put x=y=0,we get f[0]=2f[0]f[0]+2f[0] =>f[0]=0,-1/2,as f[0] is not equal to 0,f[0]=-1/2 now put y=0,we get f[x^2]=2f[x]f[0]+f[x^2]+f[0]=-f[x]+f[x^2]-1/2 =>f[x]=-1/2 but when f[x]=-1/2 ,f[x^2]+f[y^2]<f[x^2+y^2] there is no solution when f[0] is not equal to 0

GRCMIRACLES wrote: CASE-1:if f[x^2+y^2] was maximum,then f[x+y]^2=2f[x]f[y]+f[x^2+y^2] if we put x=y=0,we get f[0]=2f[0]f[0]+f[0] =>f[0]=0,so this case is not possible CASE-2:if f[x^2]+f[y^2] is maximum f[x+y]^2=2f[x]f[y]+f[x^2]+f[y^2] if we put x=y=0,we get f[0]=2f[0]f[0]+2f[0] =>f[0]=0,-1/2,as f[0] is not equal to 0,f[0]=-1/2 now put y=0,we get f[x^2]=2f[x]f[0]+f[x^2]+f[0]=-f[x]+f[x^2]-1/2 =>f[x]=-1/2 but when f[x]=-1/2 ,f[x^2]+f[y^2]<f[x^2+y^2] there is no solution when f[0] is not equal to 0 You can't do this,there might be the cases that $f(x^2+y^2)\ge f(x^2)+f(y^2)$ for only some pairs $(x,y)$ but $f(x^2+y^2)<f(x^2)+f(y^2)$ for another pairs $(x,y)$.

Let $P(x,y)$ denote the assertion of the problem. I claim that $f(x)=x-1$ and $f(x)=1$ are the only solutions. $$P(0,0) \implies -f(0)^2=\max \left\{f(0),2f(0) \right\}\implies f(0)<0\implies -f(0)^2=f(0)\implies f(0)=-1$$$$P(x,0)\implies f(x)^2=-2f(x)+\max \left\{ f(x^2),f(x^2)-1\right\}\implies 1+f(x^2)=\left(1+f(x)\right)^2 \ \ \ (1)$$Clearly $(1)$ implies that $\left(1+f(x)\right)^2=\left(1+f(-x)\right)^2$, thus $f(x)=f(-x)$ or $f(-x)=-2-f(x) \ \ \ (2)$. Case 1: There exists a real $x_0 \neq 0$, such that $f(x_0)=f(-x_0)$. Claim 1: If $f(x)=f(-x)$ then $f(-2x)=f(2x)=f(4x^2)=-1$. Suppose that $f(u)=f(-u)$, then $P(x+u,u)-P(x+u,-u)\implies f(x+2u)^2=f(x)^2 \ \ \ (3)$ , thus $f(2u)^2=f(-2u)^2=1$, from $(2)$ easily follows that $f(2u)=f(-2u)$, thus $f(4u)^2=1$. If $f(2u)=1$ then from $(1)$ we have that $f(4u^2)=3$, thus plugging in $P(2u,2u)$ gives us that $f(8u^2)=-1$, but now plugging in $P(4u^2,4u^2)$ we have that $-17=\max \left\{f(32u^4),2f(16u^4) \right\}$, which is clearly contradicting $(1)$, hence $f(2u)=f(-2u)=-1$, and it follows that $f(4u^2)=-1 \ \ \ \square$ It easily follows from Claim 1 that there exist infinitely many and arbitrarily large/small numbers $x$ which satisfy $f(x)=-1$. Claim 2: $f(x)=f(-x)$ for all real numbers $x$. Suppose that there exists a real $v$, for which $f(v)+f(-v)=-2$ and $f(v)\neq -1$, then $\max \left\{|f(v)|,|f(-v)| \right\}>1$. Assume WLOG that $\max \left\{|f(v)|,|f(-v)| \right\}=|f(v)|\stackrel{\text{def}}{=}M>1$ and define $x_{k+1}=\sqrt{2x_k^2-1}$, with $x_0=M$ then it follows from an induction that $|f(2^iv)|\ge x_i$, but because $x_n$ is unbounded, this means that $f$ can't be both upper and lower bounded, but from $(1)$ and $(2)$ it follows that $f$ is neither upper nor lower bounded, and moreover $f$ isn't upper bounded on positive reals. Now pick $u>x>0$ such that $f(x)>20$ and $f(u)=-1$, then from $P(2u-x,x)$ it follows that $f(2u-x)f(x) \le 1$, but $(3) \implies |f(2u-x)|=|f(-x)|>10 \implies f(2u-x)f(x)<0\implies f(2u-x)f(x)<-10$, contradicting $(1)$, because both $2u-x$ and $x$ are positive.$ \ \ \ \square$ Now $P(x,y)-P(x,-y)\implies f(x+y)^2=f(x-y)^2\implies f(x)^2=1\implies f(x)=-1$ for all reals $x$. Case 2: $f(x)+f(-x)=-2$ for all reals $x$. Define $g(x)=1+f(x)$ and note that $(1) \implies g(x) \ge 0$ for positive reals $x$. $$P(x,y)=P(x,-y) \implies f(x)+f(y)=f(x+y)-1 \implies g(x+y)=g(x)+g(y) \implies g(x)=cx\implies f(x)=x-1 \ \ \ \blacksquare$$

Oh, dear. The answer is $f \equiv -1$ and $f(x) = x-1$, both of which are easily seen to work. We now show necessity. Let $P(x,y)$ denote the assertion. We have the following preliminaries: Giving $P(0,0)$ yields $-f(0)^2 = \max\{f(0), 2f(0)\}$. Evidently $f(0) < 0$, so $-f(0)^2 = f(0)$ or $f(0) = -1$. Comparing $f(x,y)$ and $f(x,-y)$ yields the equivalence $$f(x+y)^2 - f(x-y)^2 = 2f(x)(f(y) - f(-y)). \qquad (\heartsuit).$$In particular, $x = 0$ yields $f(y) = f(-y)$ or $f(y) + f(-y) = -2$. Giving $P(x, 0)$ yields $$f(x)^2 + 2f(x) = f(x^2) \implies (f(x)+1)^2 = f(x^2) + 1. \qquad (\clubsuit)$$In particular, plugging in $x = 1$ yields $f(1) \in \{0, -1\}$. We now split into two cases. Case 1: $\mathbf{f(1) = -1}$. Plugging in $x = -1$ into $(\clubsuit)$ gives $f(-1) = -1$ as well. Suppose $f$ is nonconstant. Discard the case where $f$ is even everywhere; otherwise, $(\heartsuit)$ yields $f(x+y)^2 = f(x-y)^2$ or $f(x) \in \{-1,1\}$ everywhere, and $(\clubsuit)$ quickly yields $f$ constant. Moreover, comparing $\heartsuit(\tfrac12, \tfrac12)$ and $\heartsuit(-\tfrac12, -\tfrac12)$ forces $f(\tfrac12) = f(-\tfrac12)$. Then substituting $\heartsuit(x+\tfrac12,\tfrac12)$ yields $f(x)^2 = f(x+1)^2$, and using this to compare $\heartsuit(x,y)$ and $\heartsuit(x+1,y)$ yields $f(x) = f(x+1)$. (In particular, $f(n) = -1$ for all integers $n$.) Similarly, note $(f(\pm \sqrt{2})+1)^2 = f(2) + 1 = 0$, so $f(\pm \sqrt{2}) = -1$. Then, as before, $\heartsuit(x+\sqrt{2}, \sqrt{2})$ yields $f(x+2\sqrt{2})^2 = f(x)^2$, and comparing $\heartsuit(x,y)$ and $\heartsuit(x,2\sqrt{2})$ yields $f(x+2\sqrt{2}) = f(x)$. As $f$ is nonconstant, $f$ is periodic for some minimal period $t$. However, $f$ is also periodic with period $a+2b\sqrt{2}$ for any $a,b\in \mathbb{Z}$, which can get arbitrarily close to $0$, contradiction. So $f \equiv -1$ here. $\square$. Case 2: $\mathbf{f(1) = 0}$. This case requires a few more steps. Note that $(\clubsuit)$ gives $f(-1) \in \{0,-2\}$. However, if $f(-1) = 0$, then $\heartsuit(\tfrac12, \tfrac12)$ and $\heartsuit(-\tfrac12, -\tfrac12$ yield $$2f(\tfrac12)(f(\tfrac12) - f(-\tfrac12)) = -1 = 2f(-\tfrac12)(f(-\tfrac12) - f(\tfrac12)),$$which is a contradiction. So $f(-1) = -2$. Plugging in $P(1,-1)$ yields $1 = \max\{f(2),0\}$, so $f(2) = 1$. The equation $\heartsuit(x,1)$ yields $$f(x+1)^2 - f(x-1)^2 = 4f(x).$$Moreover, $(\clubsuit)$ yields $f(x) \geq -1$ for $x \geq 0$, so by induction $f(n) = n-1$ for $n \in \mathbb{N}$. Note $(\heartsuit)$ also implies $f(-n) \in \{-n-1, n-1\}$. Once again by induction, $f(-n) = -n-1$, or $f(k) = k-1$ for $k \in \mathbb{Z}$. The pith of this case is the following claim. Claim. We have $f(a) \neq f(-a)$ for $a \neq 0$. In particular, $f(a) + f(-a) = -2$. Proof. Suppose not, so $f(a) = f(-a)$ for $a \neq 0$. Then $\heartsuit(a,-a)$ yields $f(x+a)^2 = f(x-a)^2$, and comparing $\heartsuit(x,1)$ and $\heartsuit(x+2a,1)$ yields $f(x) = f(x+2a)$, or $f$ periodic for some minimal period $t$ (as $f$ is nonconstant). Then $P(x,0)$ and $P(x,t)$ yields $f(t^2) = 0$ (which is not possible by $(\clubsuit)$), or $f(x^2+t^2) = f(x^2)$, so $t = 1$. However, this is a contradiction, as $f(1) \neq f(2)$, as desired. $\square$ Now sum the expressions $P(x,y)$ and $P(x,-y)$ to get $$f(x+y)^2 + f(x-y)^2 = -4f(x) + 2\max\{f(x^2+y^2), f(x^2) + f(y^2)\}.$$In particular, replacing $x \to -x$ and using $f(a) + f(-a) = -2$ yields $$(-2-f(x-y))^2 + (-2-f(x+y))^2 = -4(-2-f(x)) + 2\max\{f(x^2+y^2), f(x^2) + f(y^2)\}.$$Subtracting yields $f(x+y) + f(x-y) = 2f(x)$, i.e. $f$ satisfies Jensen's functional equation. Thus, as $f(0) = -1$, we have $f(x) = g(x) - 1$ for additive $g$. Finally, $(\clubsuit)$ yields $g(x^2) = g(x)^2$, or $g$ bounded above on $[0,\infty)$, so $g$ is linear. Then evidently $f(x) = x-1$. Yeehaw! $\blacksquare$ Remark. Some notes on motivation: it seems like the main pothole is trying to navigate the unwieldy $\max$. This is why we mostly focus on $(\heartsuit)$ and $(\clubsuit)$ at all costs.

Wonderful problem! The problem statement is definitely one of the most unusual FEs I have seen. Anyway, here's my solution: The only solutions are $f \equiv -1$ and $f(x)=x-1$ for all $x \in \mathbb{R}$, which can easily seen to satisfy the given conditions. Now we show that these are the only solutions. Let $P(x,y)$ denote the given assertion. Then we have $$P(0,0) \Rightarrow \max \{f(0),2f(0) \}=-f(0)^2 \leq 0 \Rightarrow f(0) \leq 0 \Rightarrow f(0) \geq 2f(0)$$This gives $f(0)=-f(0)^2$, which combined with $f(0) \neq 0$ forces $f(0)=-1$. Then $$P(x,0) \Rightarrow f(x)^2+2f(x)=\max\{f(x^2),f(x^2)-1\}=f(x^2) \Rightarrow \boxed{(f(x)+1)^2=f(x^2)+1} \quad (\spadesuit)$$Putting $x \mapsto -x$ in $(\spadesuit)$, we have $$(f(x)+1)^2=(f(-x)+1)^2 \Rightarrow f(x)=f(-x) \text{ OR } f(x)+f(-x)=-2$$Also, since the expression inside the max bracket in $P(x,y)$ is even, so we get $$P(x,y)-P(x,-y) \Rightarrow \boxed{f(x+y)^2-f(x-y)^2=2f(x)(f(y)-f(-y))} \stackrel{\text{def}}{:=} Q(x,y)$$The crucial claim is the following (which kinda deals with a pointwise trap)- CLAIM Either $f(x)=f(-x)$ for all $x \in \mathbb{R}$ or $f(x)+f(-x)=-2$ for all $x \in \mathbb{R}$. (Proof) FTSOC assume there exist $a,b \in \mathbb{R}-\{0\}$ such that $f(a)=f(-a)$ and $f(b)+f(-b)=-2$ holds simultaneously. Here, we have $f(a),f(b) \neq -1$. Note that $$Q(x+a,a) \Rightarrow f(x+2a)^2=f(x)^2$$Then we get $$Q(a,x+a) \Rightarrow 2f(a)(f(x+a)-f(-x-a))=f(x+2a)^2-f(-x)^2=f(x)^2-f(-x)^2$$Notice that, for $x=b$, since $f(b) \neq \pm f(-b)$, so we get $f(b+a) \neq f(-b-a)$. Also, doing the transformation $x \mapsto x-a$, we have $$2f(a)(f(x)-f(-x))=f(x+a)^2-f(-x-a)^2$$Putting $x=b$ in the above two equalities and multiplying them together, we have $$4f(a)^2(f(b)-f(-b))(f(b+a)-f(-b-a))=(f(b)^2-f(-b)^2)(f(b+a)^2-f(-b-a)^2)$$Since $f(b+a) \neq f(-b-a)$, so we must also have $f(b+a)+f(-b-a)=-2$. Then cancelling $f(b)-f(-b)$ and $f(b+a)-f(-b-a)$ from both sides, we find that $$4f(a)^2=(f(b)+f(-b))(f(b+a)+f(-b-a))=4 \Rightarrow f(a)^2=1 \Rightarrow f(a)=1$$where the last step follows from the fact that $f(a) \neq -1$. Now, as shown before, we have $f(2a)^2=f(0)^2=1$. Also, $$P(a,a) \Rightarrow f(2a)^2=2f(a)^2+\max\{f(2a^2),2f(a^2)\} \geq 2f(a)^2+2f(a^2)$$Using $(\spadesuit)$, the above inequality translates to $$1=f(2a)^2 \geq 2f(a)^2+2(f(a)^2+2f(a)) \Rightarrow 1 \geq 4f(a)^2+4f(a)$$which is clearly false for $f(a)=1$. Thus, we arrive at a contradiction, as desired. $\Box$ Return to the problem at hand. We handle the two cases of our Claim separately:- First consider the situation when $f(x)+f(-x)=-2$ for all $x \in \mathbb{R}$. Then \begin{align*} Q(-x,y) &\Rightarrow f(x+y)^2-f(-x-y)^2 =2f(x)f(y)-2f(-x)f(-y) \\ &\Rightarrow f(x+y)^2-(-2-f(x+y))^2=2f(x)f(y)-2(-2-f(x))(-2-f(y)) \\ &\Rightarrow -4-4f(x+y)=-8-4f(x)-4f(y) \\ &\Rightarrow f(x+y)+1=(f(x)+1)+(f(y)+1) \\ \end{align*}This means that the function $f+1$ is additive. But, from $(\spadesuit)$, we already know that the function $f+1$ takes only positive values for positive reals, i.e. it is bounded from below on the set of positive reals. Thus, $f+1$ must be of the form $ax$. Then $(\spadesuit)$ gives $a^2=a$, i.e. $a=0,1$. Thus, we have the two solutions mentioned in the beginning of our solution. $\text{ }$ Now consider the case when $f$ is even. Note that $Q(x,x)$ gives $f(2x)^2=f(0)^2=1$. Suppose $f(2m)=1$ for some $m \in \mathbb{R}$. Then, from $(\spadesuit)$, we have $f(4m^2)=f(2m)^2+2f(2m)=3$. But $$P(2m,2m) \Rightarrow 1=f(4m)^2=2f(2m)^2+\max\{f(8m^2),2f(4m^2)\} \geq 2f(2m)^2+2f(4m^2)=8$$which is a contradiction. So $f(2x) \neq 1$ for any $x$, which gives the function $f \equiv -1$. $\blacksquare$

I claim that the only solutions are $f \equiv -1$ and $f(x) = x-1$ . Let $P(x,y)$ denote the assertion . Note that $$P(0,0) \implies -f(0)^2=\operatorname{max} \{f(0), 2f(0) \} \implies f(0)=-1$$. Note that $$P(x,0)\implies f(x)^2=-2f(x)+\max \left\{ f(x^2),f(x^2)-1\right\}\implies 1+f(x^2)=\left(1+f(x)\right)^2 \ \ \ (1)$$The above line gives away the fact that $$f(x)=f(-x) \text{ OR } f(x)+f(-x)=-2$$Also we consider the following assertions :- $$P(x,y)-P(x,-y) \Rightarrow \boxed{f(x+y)^2-f(x-y)^2=2f(x)(f(y)-f(-y))} \stackrel{\text{def}}{:=} Q(x,y)$$$$P(x,y)-P(-x,-y) \Rightarrow \boxed{f(x+y)^2-f(-x-y)^2=2(f(x)f(y)-f(-x)f(-y))} \stackrel{\text{def}}{:=} R(x,y)$$ Now Note that $Q(x,x) \implies f(2x)^2 = 1+2f(x)(f(x)-f(-x))$ Therefore we get that $$f(2x) = \begin{cases} \pm1 & \text{if} f(x)=f(-x) \\ \pm(2f(x)+1) & \text{if} f(x)+f(-x)=-2 \end{cases}$$ CLAIM $f(x)+f(-x)=-2$ for all $x \in \mathbb{R}$. Proof :- Suppose FTSOC that there exists a $a \in \mathbb{R}-\{0\}$ such that $f(a)=f(-a)$ and $f(a)\neq-1$ Now note that we cannot have $f(a)=1$ . Otherwise , it'd imply that $f(2a)^2=1$ , and we'd get a contradiction by $P(a,a)$ Therefore we have that the following hold $$f(a)=\pm(2f(\frac a2)+1) $$$$f(-a)=\pm(2f(\frac {-a}{2})+1) $$ Now we consider $R(\frac a2 , \frac a2) $ $$R(\frac a2 ,\frac a2) \implies f(a)^2-f(-a)^2 = 2(f\left (\frac {a}{2} \right)^2-f\left (\frac {-a}{2} \right)^2)$$ However this implies that $f(\frac a2)+f(\frac{-a}{2}) =0$ . This is because we cant have $f(\frac a2)=f(\frac{-a}{2}) $ or else $f(a)^2=1$ . Therefore we arrive at a contradiction as we wanted $f(\frac a2)+f(\frac{-a}{2}) =2$ . FInishing Now define $g=f+1$ . Note that $R(x,y)$ rearranges to give $$f(x+y)^2-2f(x)f(y) = f(-x-y)^2-2f(-x)f(-y) \implies f(x+y)=f(x)+f(y)+1$$Hence we note that $g$ is additive . Further note that $g(x^2)=g(x)^2 \geq 0 \implies \text{g is bounded}$ Hence $g(x)=cx$ for a constant $c \in \mathbb{R}$ . Plugging back we note that only $c=0$ and $c=1$ works . This gives us our solutions

Denote the assertion as $P(x,y)$. $P(0,0)$ gives $f(0)^2=-\max\{f(0),2f(0)\}$, which gives $f(0)=0,-1,-2$. Out of these, only $f(0)=-1$ works. $P(x,0)$ gives $f(x)^2+2f(x)=f(x^2)\,\,(\star)$ Now, by $(\star)$, $f(x),f(-x)$ are the solutions of the quadratic equation $r^2+2r-f(x^2)=0$, so either $f(x)=f(-x)$ or $f(x)+f(-x)=-2$. Suppose we have nonzero $k$ such that $f(k)=f(-k)$. Then, note that the value of the $\max$ is the same in both $P(x,k)$ and $P(x,-k)$, so we actually have $f(x+k)^2=f(x-k)^2$ after subtracting them. In particular, this means that $f(2kn)=\pm 1$ for all $n\in\mathbb{Z}$. Now, suppose there exists $n_0$ such that $f(2kn_0)=-1$. Let $M=2kn_0$. We either have $f(-M)=-2-f(M)=-1$ or $f(-M)=f(M)=-1$, so either way $f(M)=f(-M)=-1$. So, WLOG $M>0$. Like before, $f(x+2M)=\pm f(x)$ for all $x$. Now, by $(\star)$, we have that $f(\pm \sqrt{M})=-1$, so by similar logic, $f(2\sqrt{M})=\pm 1$ and $f(x+2\sqrt M)=\pm f(x)$. If $f(2\sqrt M)=1$, consider $P(x,2\sqrt M)$. We get $f(x)^2=2f(x)+\max\{f(x^2+4M),f(x^2)+f(4M)\}$. For the max, we have 4 possible cases of what it is. In particular, it can be $\pm f(x^2)$ or $f(x^2)\pm 1$. However, if we let $x=0$, we need the max to be $f(0)^2-2f(0)=3$, and its only possible values are $-1,0,1,2$, which is a contradiction. Hence, $f(2\sqrt M)=f(-2\sqrt M)=-1$. Hence, $f(4M)=-1$ by $(\star)$, and $P(x,2\sqrt M)$ now gives $f(x)^2-2f(x)=\max\{f(x^2)\pm 1, f(x^2+4M)\}\implies f(x^2+4M)=f(x^2)$, and we must have $f(x+4M)=f(x)$ for all $x\ge 0$. Using $P(0,y)$ and $P(0,y+4M)$, the non max parts are now identical, so we end up with $$\max\{f(y^2),f(y^2)-1\}=\max\{f((y+4M)^2),f((y+4M)^2)-1\}\implies f(y^2)=f(y^2+8yM+16M^2)$$Now, as $y$ varies, the quantity $8yM+16M^2$ will range over all reals modulo $4M$, meaning that we have $f(x)$ is constant for all large $x\ge C$ for some large $C$. $f(2Mn)=\pm 1$ for all $n\in\mathbb Z$, so the constant is $\pm 1$, and plugging them both into $P(x,y)$ when $x,y\gg C$ shows that only $-1$ works. Now, $P(x,y)$ for $x\gg 2020C$ gives $1+2f(y)=\max\{-1,f(y^2)-1\}$. If the max yields $-1$, $f(y)=-1$, and otherwise we have by $(\star)$ that $f(y)^2=2$. So, $f(x)=-1,\pm\sqrt 2$ for all $x$. If $f(d)=\pm\sqrt 2$, then $(\star)$ gives $f(d^2)=2\pm 2\sqrt 2$, which is not in the range. Hence, $f(x)\equiv -1$. Now, if there does not exist such $n_0$, then $f(2kn)=1$ for all $n\in\mathbb Z$, and $(\star)$ gives $f(4n^2k^2)=3$. Considering $P(6k, 8k)$, we get $$f(14k)^2=2f(6k)f(8k)+\max\{f(100k^2),f(36k^2)+f(64k^2)\}\implies 1=2+6$$which is a contradiction. Hence, $k$ does not exist and $f(x)+f(-x)=-2\forall x$. Consider $P(x,y)$ and $P(-x,-y)$. Their maxes yield the same value, so if we subtract the two we get that $$(-2-f(x+y))^2-f(x+y)^2=2((-2-f(x))(-2-f(y))-f(x)f(y))\implies f(x+y)=f(x)+f(y)+1$$For convenience, define $g(x)=f(x)+1$, so this becomes $g(x+y)=g(x)+g(y)$, and $(\star)$ translates into $g(x^2)=g(x)^2$. $g(x+y)=g(x)+g(y)$ is Cauchy, and $(\star)$ tells us that it is bounded below by $0$ over the positives. Hence, the solution cannot be pathological and $g(x)=cx$. $(\star)$ gives that only $c=0,1$ work, so $g(x)\equiv0$ or $g(x)=x$. Having exhausted all cases, we must either have $f(x)\equiv -1$ or $f(x)=x-1$.

Nice problem, one of the most fun to work on FEs I've done. We claim that the only solutions are $f(x)\equiv -1,x-1$, both of which can easily be seen to work. Now suppose some function $f$ works. Let $P(x,y)$ denote the given FE. We see that \[P(0,0)\implies f(0)^2 = 2f(0)^2 + \max\{f(0),2f(0)\}.\]It is easy to see that this implies that $f(0)\in\{0,-1\}$, and the problem statement then tells us that $f(0)=-1$. We now have that \[P(x,0)\implies f(x)^2 = -2f(x)+f(x^2),\]which simplifies to \[[f(x)+1]^2 = f(x^2)+1.\quad\quad\quad(\star)\]Replacing $x\to -x$ in $(\star)$, we see that \[f(x) = f(-x)\quad\quad\text{or}\quad\quad f(x)+f(-x) = -2.\quad\quad\quad(\star\star)\]It is important to note that by $(\star\star)$, \[f(x)=-1\iff f(-x)=-1.\]Comparing $P(x,y)$ and $P(x,-y)$, we get the equation \[Q(x,y):\quad f(x+y)^2 - f(x-y)^2 = 2f(x)[f(y)-f(-y)].\]Comparing $P(x,y)$ and $P(-x,-y)$, we get the equation \[R(x,y):\quad f(x+y)^2 - f(-x-y)^2 = 2f(x)f(y)-2f(-x)f(-y).\]Armed with these four equations, we can start making some headway. Claim: If $f$ is even, then $f\equiv -1$. Proof: We see that \[Q(x,y)\implies f(x+y) = \pm f(x-y)\]given that $f$ is even, so $f(x)\in\{f(0),-f(0)\}$, so $f(x)\in\{-1,1\}$ for all $x$. Now, if $f(x)=1$ for some $x$, then by $(\star)$, we see that \[f(x^2) = -1 + 2^2 = 3,\]which is not in $\{-1,1\}$, which is a contradiction. Thus, $f(x)=-1$ for all $x$, as desired. $\blacksquare$ Now, we assume that $f$ is not even, so in particular, we have some $\alpha$ such that $f(\alpha)\ne f(-\alpha)$. We now have the following key lemma. Lemma: If $f(z)=f(-z)$ for some $z$, then $f$ is periodic with period $2z$. Proof: We see that \[Q(x,z)\implies f(x+z)^2 = f(x-z)^2\]for all real $x$, so we have that \[f(x+2z)=\pm f(x)\]for all real $x$. It suffices to pin down the sign now. This uses $\alpha$. We see that \begin{align*} Q(x,\alpha) &\implies f(x+\alpha)^2 - f(x-\alpha)^2 = 2f(x)[f(\alpha)-f(-\alpha)] \\ Q(x+2z,\alpha) &\implies f(x+\alpha)^2 - f(x-\alpha)^2 = 2f(x+2z)[f(\alpha)-f(-\alpha)], \end{align*}and comparing these two equations along with $f(\alpha)-f(-\alpha)\ne 0$ gives us $f(x+2z) = f(x)$ for all real $x$. This completes the proof of the lemma. $\blacksquare$ We see that $(\star)$ implies $[f(1)+1]^2 = f(1)+1$, so $f(1)\in\{0,-1\}$. Claim: If $f(1)=-1$, then $f\equiv -1$. (Logically, since we've assumed $f$ is not even, this claim could alternatively be phrased as $f(1)\ne -1$.) Proof: Let $x$ be some real number in the closed interval $[0,1]$. The main claim is that $f(2x) = f(2(1-x)) = -1$. Suppose first that $f(x) = f(-x)$. Then, $f$ is periodic with period $2x$, so $f(2x)=f(-2x)=-1$. However, $f(1)=-1$, so $f(1)=f(-1)$, so $f$ is periodic with period $2$ as well, so $f(2-2x) = f(-2x)=-1$, so \[f(2x) = f(2-2x) = -1,\]as desired. The analysis of the case $f(1-x) = f(x-1)$ is similar, so we may now assume that $f(x)\ne f(-x)$ and $f(1-x)\ne f(x-1)$. Thus, $f(-x) = -2 - f(x)$ and $f(x-1) = -2 - f(1-x)$, so \[R(x,1-x)\implies f(x)f(1-x) = f(-x)f(x-1)\]becomes \[f(x) + f(1-x)=-2\]upon expansion. But, $x,1-x\ge 0$, and $f(t^2) = -1+[f(t)+1]^2\ge -1$, so $f(x),f(1-x)\ge -1$. Thus, the only possibility is that $f(x)=f(1-x) = -1$. Then, $f$ is periodic with periods $2x$ and $2(1-x)$, so $f(2x) = f(2(1-x)) = -1$, as desired. Thus, we have that $f(t)=-1$ for all $t\in[0,2]$. We know that $f$ is periodic with period $1$, so this implies that $f\equiv -1$, as desired. $\blacksquare$ We may now assume that $f(1)\ne -1$, so $f(1)=0$. In this case, we actually have the following nice reduction of $(\star\star)$. Claim: If $f(1)=0$, then we have $f(x)+f(-x) = -2$ for all real $x$. Proof: Say we have some real $x$ such that $f(x) = f(-x)$. It suffices to show that $f(x)=-1$. Indeed, \[R(x/2,x/2)\implies f(x/2)^2 = f(-x/2)^2.\]If $f(x/2) = f(-x/2)$, then $f$ has period $x$, so $f(x) = f(0) = -1$, so we're done. Thus, we may assume that $f(x/2)\ne f(-x/2)$, so $f(x/2) + f(-x/2) = -2$ by $(\star\star)$. But the displayed equation above implies that $f(x/2)+f(-x/2)=0$, which is a contradiction. Thus, $f(x)=-1$, so $f(-x)=-1$, so $f(x)+f(-x) = -2$. $\blacksquare$ Now, $R(x,y)$ simplifies to \[(-2)[2f(x+y) + 2] = 2[f(x)f(y) - (-2-f(x))(-2-f(y))],\]which becomes \[f(x+y)+1 = [f(x)+1]+[f(y)+1]\]for all real $x,y$. We also have that $f(x)+1\ge 0$ for $x\ge 0$ by $(\star)$, so $f+1$ is Cauchy and bounded, so it is linear. By plugging in $x=1$, we find that $f(x)+1=x$, so $f(x)=x-1$ for all real $x$, which shows that $f\equiv x-1$. Thus, in all cases, $f\equiv -1$ or $f\equiv x-1$, so we're done.

The answer is $f(x)\equiv x-1, f(x)\equiv -1$. They clearly work. $P(0,0)$ yields $-f(0)^2=\max\{f(0),2f(0)\}$, which implies $f(0)<0,$ so $f(0)=-1$. The key idea is that $\max\{ f(x^2+y^2), f(x^2)+f(y^2)\}$ is a very strange, hard term to deal with, but this term is same if I negate $x$ or $y$, so it's a good idea to compare $P(\pm x, \pm y)$ First, $P(x,0)$ gives $f(x)^2=-2f(x)+\max\{f(x^2),f(x^2)-1\}$, or $(f(x)+1)^2=1+f(x^2)=(f(-x)+1)^2$. Therefore, either for any $x\in \mathbb{R}, f(x)=f(-x)$ or $f(x)+f(-x)+2=0$ Let $S=\{ x: f(x)=f(-x)\}$. Case 1: $S$ is nonempty. Suppose $x\in S, $ then $P(x,x),P(x,-x), P(-x,-x)$ yields $f(2x)^2=1=f(-2x)^2$. Since $(f(2x)+1)^2=(f(-2x)+1)^2,$ it follows that $f(2x)=f(-2x)$, so $2x\in S$. $P(x,y)-P(x,-y): f(x+y)^2-f(x-y)^2=2f(x)(f(y)-f(-y))=2f(x)f(y)-2f(x)f(-y). P(x,y)-P(-x,y): f(x+y)^2-f(y-x)^2=2f(y)f(x)-2f(y)f(-x)$ Therefore, $(P(x,y)-P(x,-y))-(P(x,y)-P(-x,y)): f(y-x)^2-f(x-y)^2=2f(x)f(-y)-2f(-x)f(y)$. Let $d=y-x$, then $f(d)^2-f(-d)^2=2f(x)f(-x-d)-2f(-x)f(x+d)$. I claim $S=\mathbb{R}$. Assume there exist $d$ such that $f(d)\ne f(-d)$. Suppose $x\in S$. Since $f(x)=f(-x),$ $(f(d)-f(-d))(-2) = 2f(x)(f(-x-d)-f(x+d))$, so it follows that $f(-x-d)-f(x+d)\ne 0$ as well. Rewriting, $2(2f(d)+2)=2f(x)(2f(x+d)+2),$ or $f(d)+1=f(x)(f(x+d)+1)$ Let $g(x)=f(x)+1$. This implies $g(d)=f(x)g(x+d)$, and $g(x+d)=f(-x)g(d)$. Since $g(d)\ne 0$ as $f(d)\ne f(-d)$, it follows that $g(x+d)\ne 0,$ so $f(x)f(-x)=1$ for all $x$. Now, $P(x,-x)$ yields $1=2+\max\{f(2x^2),2f(x^2)\}$. Therefore, $\max\{f(2x),2f(x)\}=-1$ for all $x\ge 0$. Suppose $f(2x)<-1$, then $2f(x)=-1,$ so $f(x)=\frac{-1}{2}$. Since $f(x)f(-x)=1, f(-x)=2$. However, $(f(x)+1)^2=\frac 14 \ne 1 = (f(-x)+1)^2$, contradiction. Therefore, $f(x)=-1$ for all $x\in \mathbb{R}_{\ge 0}$. Since $f(x)=f(-x), f(x)\equiv -1$. Case 2: $S$ is empty. This implies $f(x)+f(-x)+2=0$ is an identity. $P(x,y)-P(-x,y)$ gives $f(x+y)^2-f(x-y)^2=2f(x)f(y)-2f(x)f(-y)=2f(x)(f(y)-f(-y))=2f(x)(2f(y)+2)=4f(x)(f(y)+1)$. Swap $x,y$, we can see $f(x+y)^2-f(y-x)^2=4f(y)(f(x)+1)$, so $f(y-x)^2-f(x-y)^2=4(f(x)-f(y))$ However, $f(y-x)^2-f(x-y)^2=(f(y-x)+f(x-y))(f(y-x)-f(x-y))=(-2)(2f(y-x)+2)=4(f(x)-f(y))$ This rewrites to $f(y-x)+1=f(y)-f(x)$ Rewriting to the original equation, $2f(x)f(y)+f(x^2+y^2)=f(x+y)^2=(f(x)+f(y)+1)^2=f(x)^2+f(y)^2+2f(x)f(y)+2f(x)+2f(y)+1$ Hence, $f(x^2)+f(y^2)=f(x)^2+f(y)^2+2f(x)+2f(y)$ Let $g(x)=f(x)+1$, then $g(x^2)+g(y^2)=g(x)^2+g(y)^2$. Since $g(0)=0, g(x^2)=g(x)^2$. Since $g$ is Cauchy and increasing, $g$ is linear and identity.

Let $P(x,y)$ be the assertion $f(x+y)^2=2f(x)f(y)+\max\{f(x^2+y^2),f(x^2)+f(y^2)\}$. $P(0,0)\Rightarrow f(0)^2+\max\{f(0),2f(0)\}=0$ If $f(0)>0$, then $0=f(0)^2+2f(0)>0$, contradiction. Thus $\max\{f(0),2f(0)\}=f(0)$, so $f(0)=-1$. $P(x,0)\Rightarrow f(x)^2+2f(x)=f(x^2)\Rightarrow f(x)^2+2f(x)=f(-x)^2+2f(-x)$ Let $A=\{x\in\mathbb R\mid f(x)=f(-x)\}$ and $B=\{x\in\mathbb R\mid f(-x)=-f(x)-2\}$. This implies $A\cup B=\mathbb R$. Also, if $f(x)^2=f(-x)^2$, then $f(x)=f(-x)$. Suppose that there is an $a\in A$ and a $b\in B$ with $-1\notin\{f(a),f(b)\}$. $P(a,b)\Rightarrow f(a+b)^2=2f(a)f(b)+\max\{f(a^2+b^2),f(a^2)+f(b^2)\}$ $P(-a,-b)\Rightarrow f(-a-b)^2=-2f(a)f(b)-4f(a)+\max\{f(a^2+b^2),f(a^2)+f(b^2)\}$ Case 1: $a+b\in A$ Then $f(a)=0$ from comparing $P(a,b)$ and $P(-a,-b)$ and using that $f(b)\ne-1$. $P(a,0)\Rightarrow f(a^2)=0$ $P(x,a)\Rightarrow f(x)^2=\max\{f(x^2+a^2),f(x^2)\}$ Thus $f(-x)^2=f(x)^2$ for all $x$, so $A=\mathbb R$, contradiction. Case 2: $a+b\in B$ Comparing the two assertions, $f(a+b)+f(a)f(b)+f(a)+1=0$. We do the same changing $(a,b)$ with $(a,a+b)$. If $2a+b\notin B$, we have a contradiction by Case 1. Thus $2a+b\in B$. $P(a,a+b)\Rightarrow f(2a+b)^2=2f(a)f(a+b)+\max\{f(2a^2+2ab+b^2),f(a^2)+f(a^2+2ab+b^2)\}$ $P(-a,-a-b)\Rightarrow f(-2a-b)^2=2f(-a)f(-a-b)+\max\{f(2a^2+2ab+b^2),f(a^2)+f(a^2+2ab+b^2)\}$ Comparing, $f(a)^2f(b)+f(a)^2=f(2a+b)+1$. $P(a,a)\Rightarrow f(2a)^2=2f(a)^2+\max\{f(2a^2),2f(a^2)\}$ $P(-a,-a)\Rightarrow f(-2a)^2=2f(a)^2+\max\{f(2a^2),2f(a^2)\}$ Thus $2a\in A$. $P(2a,b)\Rightarrow f(2a+b)^2=2f(2a)f(b)+\max\{f(4a^2+b^2),f(4a^2)+f(b^2)\}$ $P(-2a,-b)\Rightarrow f(-2a-b)^2=2f(2a)(-f(b)-2)+\max\{f(4a^2+b^2),f(4a^2)+f(b^2)\}$ Thus $f(2a+b)+f(2a)f(b)+f(2a)+1=0$. From $f(a)^2f(b)+f(a)^2=f(2a+b)+1$, we have $f(a)^2f(b)+f(a)^2+f(2a)f(b)+f(2a)=0$. $P(a,-a)\Rightarrow 1=2f(a)^2+\max\{f(2a^2),2f(a^2)\}$ $P(a,a)\Rightarrow f(2a)^2=2f(a)^2+\max\{f(2a^2),2f(a^2)\}$ So $f(2a)^2=1$. Consider $f(a)^2f(b)+f(a)^2+f(2a)f(b)+f(2a)=0$. Case 2.1: $f(2a)=1$ Then $f(a)^2f(b)+f(a)^2+f(b)+1=0$, so $(f(a)^2+1)(f(b)+1)=0$. But since $f(b)\ne-1$ and $f(a)^2+1>0$ we have a contradiction. Case 2.2: $f(2a)=-1$ Our equation is $f(a)^2f(b)+f(a)^2-f(b)-1$, which factors as $(f(a)^2-1)(f(b)+1)=0$. Thus $f(a)^2=1$. Since $f(a)\ne-1$, we have $f(a)=1$. $P(a,0)\Rightarrow f(a^2)=3$ $P(a,a)\Rightarrow 0=\max\{f(2a^2),6\}+1\ge7$ This is the final contradiction. So either $A=\mathbb R$ or $B=\mathbb R$. Case 1: $A=\mathbb R$ $P\left(\frac{x+y}2,\frac{x-y}2\right)\Rightarrow f(x)^2=f(y)^2$ Thus $f(x)\in\{-1,1\}$ for all $x$ by setting $y=0$. Assume that there are $a,b$ such that $f(a)=-1$ and $f(b)=1$. $P(a,b)\Rightarrow\max\{f(a^2+b^2),f(a^2)+f(b^2)\}=3$, contradiction. Since $f(0)=-1$, we have $\boxed{f(x)=-1}$ which is a solution. Case 2: $B=\mathbb R$ Let $g(x)=f(x)+1$. $P(-x,-y)\Rightarrow g(x+y)=g(x)+g(y)$ $P(x,0)\Rightarrow g(x^2)=g(x)^2$ Thus $g(x)\ge0$ for all $x\ge0$, so by Cauchy's FE we have $g(x)=cx$. Testing, we obtain the solution $\boxed{f(x)=x-1}$.

mathwizard888 wrote: Find all functions $f:\mathbb{R}\rightarrow\mathbb{R}$ such that $f(0)\neq 0$ and for all $x,y\in\mathbb{R}$, \[ f(x+y)^2 = 2f(x)f(y) + \max \left\{ f(x^2+y^2), f(x^2)+f(y^2) \right\}. \] ZFW Fe for OTIS!! Let \(P(x,y)\) assert the given functional equation. \(P(0,0)\) gives us that \[-f(0)^2=\max{f(0), 2f(0)}\]implying that \(f(0)\) is negative, and so it is \(-1\). Now, \(P(x,0)\) tells us that \[f(x^2)=f(x)^2+2f(x)\]Changing \(x\) to \(-x\) in this gives us that \(f(-x)\) is either \(f(x)\) or \(-2-f(x)\). Now, we take two cases. Case 1. When \(f\) is even. If \(f\) is even, then comparing \(P(x,y)\) and \(P(x,-y)\) gives us that \(f(x+y)^2=f(x-y)^2\) for all reals \(x\) and \(y\). Therefore, \(f(x)^2\) is constant. Since \(f(0)=-1\), \(f(x)^2=1\) for all reals \(x\). Now, from \[f(x^2)=f(x)^2+2f(x)=1+2f(x)\]we see that if \(f(x)=1\), then \(f(x^2)=3\), impossible. Therefore \(f\equiv-1\) for all reals \(x\). Case 2. \(f\) is not even. This means that there exists a \(x_0\) such that \(f(-x_0)=-2-f(x_0)\). We prove a claim. Claim 1. \(f(x)=-f(-x)-2\) Proof. Firstly, let \(Q(x,y)\) denote \(P(x,y)-P(-x,-y)\). Also, let \(f(t)=f(-t)\) for some real \(t\). Then, \(Q(t, y)\) tells us that \[f(y-t)-f(t-y)=f(t)(f(y)-f(-y))\]And \(Q(t,y-t)\) gives us that \[f(y)-f(-y)=f(t)(f(y-t)-f(t-y))\]implying that \[f(t)^2(f(y-t)-f(t-y))=f(y-t)-f(t-y)\]and so \(f(t)^2=1\). Assume for the sake of contradiction, that \(t=1\). Then, \(P(t,-t)\) gives us a contradiction. $\blacksquare$ Now, \(f(x)+f(-x)=-2\) for all reals \(x\), so putting that in \(Q(x,y)\) gives us that \(f(x+y)=f(x)+f(y)+1\), so \(f(x)+1\) is additive. but as \(f(x^2)+1=(f(x)+1)^2\), \(f+1\) is bounded above an interval, and hence \(f\) is linear. Therefore \(f\) is linear and checking gives us that \(f(x)=x-1\) which works. So the only solutions are \(f\equiv-1\) and \(f(x)=x-1\).

Let $P(x,y)$ be the given assertion. $P(0,0)$ gives: $$-f(0)^2= \max \{f(0), 2f(0)\}\Rightarrow f(0)=-1.$$From $P(x,0):$ $$f(x)^2+2f(x)=f(x^2)$$ Take $x\rightarrow -x$: $$f(-x)^2+2f(-x)=f(x^2)=f(x)^2+2f(x)$$$$(f(x)-f(-x))(f(x)+f(-x)+2)=0$$Case 1: We can find $f(y)=f(-y)$. Take $P(x,-y):$ $$f(x-y)^2=2f(x)f(y)+\max \{f(x^2+y^2), f(x^2)+f(y^2)\}=f(x+y)^2$$Put $x\rightarrow x+y:$ $$f(x)^2=f(x+2y)^2$$$$f(2y)^2=1$$Take $x\rightarrow -y:$ $$f(-2y)^2=1=f(2y)^2\Rightarrow f(2y)=f(-2y)$$Hence $$f(-x)^2=f(2y-x)^2=f(2y+x)^2=f(x)^2\Rightarrow f(x)=f(-x), \forall x \in \mathbb{R}.$$Using that $f(x)^2=f(x+2y)^2$, we can rewrite it as $f(x)^2=f(y)^2=1,\forall x,y \in \mathbb{R}$. Since $$\max \left\{ f(x^2+y^2), f(x^2)+f(y^2) \right\}<3$$then we can only have $\boxed{f \equiv -1.}$ Case 2: $f(x)=-2-f(-x),\forall x\in \mathbb{R}$ Take $x\rightarrow -y:$ \begin{align*} f(x-y)^2 &=-2f(x)(2-f(y))+\max \left\{ f(x^2+y^2), f(x^2)+f(y^2) \right\} \\ &=-4f(x)-2f(x)f(y)+\max \left\{ f(x^2+y^2), f(x^2)+f(y^2) \right\} \\ &=-4f(x)-4f(x)f(y)+f(x+y)^2 \end{align*}Now swap $x$ and $y$, use that $f(x-y)=-2-f(y-x)$, and get $$f(x+y)=f(x)+f(y)+1, \forall x,y \in \mathbb{R}.$$Finally, consider $g:\mathbb{R}\rightarrow \mathbb{R}, g(x)=f(x)+1\Rightarrow g(x+y)=g(x)+g(y)$, and, from $P(x,y)$, $g(x)\ge 0, \forall x\ge 0$. Thus $g(x)=c\cdot x\Rightarrow f(x)=c\cdot x-1$. After the verification, obtain: $$\boxed{f(x)=x-1, \forall x\in \mathbb{R}.}$$ In conclusion, our soulutions are $f\equiv -1$ and $f(x)=x-1, \forall x \in \mathbb{R}.$

Everything is of course easier in retrospect. $(0, 0)$ gives $f(x) = -1$ $(x, 0)$ gives $f(x)^2+2f(x) = f(x^2)$. Note that this means $f(x) \geq -1$ for positive $x$. Combining $(x, 0)$ and $(-x, 0)$ gives either $f(x) = -f(-x)$ or $\frac{f(x)+f(-x)}{2} = -1$ Note: $f(a) = -f(-a)$ for some $a > 0$ then combining $(a, y)$ and $(-a, y)$ gives that $f^2(x)$ is periodic with period $2a$. Lemma: If $\frac{f(x)+f(-x)}{2} = -1$, then $\frac{f(kx)+f(-kx)}{2} = -1$ for all integers $k$. Proof: Just induct with casework using the equation $f(kx+x)^2-f(kx-x)^2 = 2f(kx)(f(x)-f(-x))$. Hence if $f(x) = -f(-x)$ and $\frac{f(x)+f(-x)}{2} \neq -1$ then $\frac{f(\frac{x}{4})+f(-\frac{x}{4})}{2} \neq -1$ so $f(\frac{x}{4}) = f(-\frac{x}{4})$ so the function's magnitude is periodic with period $\frac{x}{2}$, so from $(-\frac{x}{2}, \frac{x}{2})$ we know that $f(\frac{x^2}{2}) = -1$ but this means from $(\frac{x^2}{2}, \frac{x^2}{2})$ that $f(x^2) = -1$ so $f(x) = -1$, so $\frac{f(x)+f(-x)}{2} = -1$, giving contradiction. Therefore $f(x) = -2 - f(-x)$. Now, combining $(x, y)$ and $(-x, -y)$ gives that $f(x+y)^2 - 2f(x)f(y) = (f(x+y)+2)^2 - 2(f(x)+2)(f(y)+2)$ which simplifies to $f(x+y) = f(x)+f(y)+1$. Hence let $f(x) = g(x) - 1$, then $g(x)+g(y) = g(x+y)$ and $g(x^2) = g(x)^2$. Since $g$ satisfies Cauchy's functional equation and is bounded below by $0$ for positive numbers, it must be linear, and it is easy to see that $cx^2 = (cx)^2$ only holds in general for $c = 0, 1$. It is easy to see that the corresponding $f(x) = -1$ and $f(x) = x-1$ both work.

Let $P(x,y)$ be the assertion. $P(0,0)\implies f(0)^2=2f(0)^2+max{2f(0),f(0)}.$ If $f(0)<0,$ then $f(0)=-1.$ Easy to see the other case has no solutions. $P(x,0)\implies f(x)^2=-2f(x)+f(x^2).~~~~~(\alpha)$ $P(-x,0)\implies f(-x)^2=-2f(-x)+f(x^2).$ Combining yields $f(x^2)+2f(x)=f(-x^2)+2f(-x).$ It follows that $f(x)=f(-x)$ or $f(x)+f(-x)=-2. ~~~~~(*)$ Combine $P(y,y)$ and $P(-y,y)$ to get $f(2x)^2-2f(x)^2=1-2f(x)f(-x). ~~~~~(\beta)$ It follows that either $f(2x)=\pm 1$ or $f(2x)=\pm(2f(x)+1).$ Claim: $f(x)+f(-x)=-2.$ Proof. Assume $\exists k: \in \mathbb{R}: f(k)+f(-k)\neq -2 \implies f(k)=f(-k)\neq -1.$ WLOG $k>0$ and assume $f(k)=1.$ $y\to k$ in $(\beta) \implies f(2k)^2=1.$ $P(k,k)\implies 1=2+max\{2f(k^2),f(2k^2)\}.$ But by $(\alpha)$ we get $f(k^2)=3,$ which contradicts our assumption. So $f(k)=\pm (2f(k/2)+1)$ and $f(-k)=\pm (2f(-k/2)+1).$ Easy to get contradiction with $f(k/2)=f(-k/2).$ So by $(*)$, $f(k/2)+f(-k/2)=-2 \implies \pm (2+f(k/2)+1)=\pm (-2f(k/2)-3),$ we can easily get contradiction with any signs. It follows that such a $k$ doesn't exist. $\blacksquare$ Combining $P(x,y)$ and $P(-x,-y)$ implies $f(x+y)^2-2f(x)f(y)=f(-x-y)^2-2f(-x)f(-y) \implies f(x+y)=f(x)+f(y)+1.$ Note that $(f(x)+1)^2=f(x^2)+1.$ Letting $g(x)=f(x)+1 \implies g(x+y)=g(x)+g(y).$ But $g(x)^2=g(x^2)$ implies $g$ is bounded. Thus by Cauchy's Functional Equation, $g(x)=cx.$ Checking we can get $c=0$ or $c=1.$ Thus $f(x)=-1~~ \forall x\in \mathbb{R} \text{ or } f(x)=x-1~~\forall x\in \mathbb{R},$ obviously both fit.

Annoying problem, I will sketch the solution. Call the assertion $P(0,0)$ it is easy to see $f(0)=-1$, then notice $f(x,0)$ note that $f(-x) \in \{f(x),2-f(x)\}$ If $f$ is even, it is clear that $f \equiv -1$ If $f \not \equiv -1$, we show by comparing $P(x,y),P(-x,-y)$ conclude that $f(-x)=2 -f(x)$ for all $x \in \mathbb{R}$ Put $g(x) =1+f(x)$ and notice that $g(x^2) \geq 0$ and $g(x+y)=g(x)+g(y)$ so $g$ is linear, so plug back check $g$ is identity so, $f \equiv x-1$ which clearly works.