We'll show that $f(x)=0$ for all $x\in \mathbb{R}$ and $f(x)=h(x)x$ for all $x\in \mathbb{R}$ for any function $h:\mathbb{R} \to \{ -1,1\}$ are the only two solutions (they clearly satisfy the condition of the problem). First we define $g:\mathbb{R}\to \mathbb{R}$ by $g(x)=f(x)^2$ for all $x\in \mathbb{R}$. Note that $g(x)\geqslant 0$ for all $x\in \mathbb{R}$. Also, we have $$P(x,y):g(g(x)-y^2)+g(2xy)=g(x^2+y^2)$$for all $x,y\in \mathbb{R}$. $P(x,y)$ and $P(x,-y)$ gives us $g(x)=g(-x)$ for all $x\in \mathbb{R}$. $P(x,x)$ then gives us $g(g(x)-x^2)=0$ for all $x\in \mathbb{R}$. If $f(r)\neq 0$ for all real number $r\neq 0$, we will get that $g(x)-x^2=0$ for all $x\in \mathbb{R}$, finishing the problem. So, suppose there exists (WLOG positive) real number $r$ that $g(r)=0$. From $P(x,y)$, we get that $g(g(x)-y^2)+g(2xy)=0$ for all $x,y\in \mathbb{R}$ that $x^2+y^2=r$. Note that the above implies $g(2xy)=0$ for all such $x,y$, which gives $f(z)=0$ for all $0\leqslant z\leqslant r$. For any positive real $N$, there exists $\delta (N)$ such that for all $0\leqslant \epsilon \leqslant \delta (N)$, $P(\epsilon ,y)$ gives us $$g(-y^2)+0=g(\epsilon^2+y^2)\implies g(y^2)=g(y^2+\epsilon^2)$$for all $0\leqslant y\leqslant N$. Using this fact, it is not hard to deduce that $g(a)=g(b)$ for any two non-negative real numbers $a,b$. So, combine with $g(x)=g(-x)$ for all $x\in \mathbb{R}$, we get that $g$ must be constant function. Plugging in $P(x,y)$ gives $g(x)=0$ for all $x\in \mathbb{R}$ and so $f(x)=0$ for all $x\in \mathbb{R}$.

@above Am I missing something, because $f(x)=x$ also works...

Mr Dark Lord wrote that $f(x)=h(x)x$ is a solution where $h:\mathbb{R} \to \{ -1,1\}$

$f(x) = x$ is also one of the above (using the case $h(x) = 1$ for all $x\in \mathbb{R}$).

Here is my solution: Let $P (x,y) $ be the given equation.One can get that $f (f (x)^2-x^2)=0$ , $f (x)^2=f (-x)^2$ and if $f (a)=0$ then $f (a^2)=0$. $\textbf {The main claim:}$If $f (a)=0$, (where $a> 0$ then $f (x)=0$ for any $-a\leq x\leq a $. $\textbf {Proof:}$ Fix $x $ and take $y $ so that $x^2+y^2=a $ $\implies $ $f (2xy)=0$ but in fact one can show by looking at the discriminant of quadratic equation that $2xy $ can get any value between $-a $ and $a $. Then if $a=f (x)^2-x^2>1$ ,so $f (t^{2^k})=0$ and by the above claim we get that $f\equiv 0$. We also have that $f (x^2-f (x)^2)=0$,so if $b=x^2-f (x)^2 >1$, then $f (b^{2^k})=0$ and similarly by the main claim we get $f\equiv 0$.So we have that $f (x)^2\leq x^2+1$,$x^2\leq f (x)^2+1$. $(x^2+y^2)^2-1\leq f (x^2+y^2)^2=f (f (x)^2-y^2)^2+f (2xy)^2\leq (f (x)^2-y^2)^2+1 + 4x^2y^2+1$ $\implies $ $(x^2-f (x)^2)(2y^2-x^2-f (x)^2)+3\ge 0$ Now, fix $x $ and look at this as a polynomial in $y $, whose leading coefficient is $x^2-f (x)^2$ so it must be positive otherwise selecting $y $ sufficiently large we get a contradiction. So $f (x)^2 \leq x^2$. $(x^2+y^2)^2+1\geq f (x^2+y^2)^2=f (f (x)^2-y^2)^2+f (2xy)^2\geq (f (x)^2-y^2)^2-1 + 4x^2y^2-1$ $\implies $ $(f (x)^2-x^2)(2y^2-x^2-f (x)^2)+3\geq 0$ similary as above we get $f (x)^2\geq x^2$ which gives $f (x)^2=x^2$ $\implies $ ${|f (x)|}={|x|}$ which one can check that work. So the solutions are $f\equiv 0$ and ${|f (x)|}={|x|}$.

Dadgarnia wrote: Find all functions $f:\mathbb{R}\rightarrow \mathbb{R}$ such that for all $x,y\in \mathbb{R}$: $$f\left(f(x)^2-y^2\right)^2+f(2xy)^2=f\left(x^2+y^2\right)^2$$ Proposed by Ali Behrouz - Mojtaba Zare Bidaki Answer: $|f(x)| = |x|$ and $f(x) \equiv 0$. Define the function $g : \mathbb{R} \rightarrow \mathbb{R}$ such that $g(x) = f(x)^2 \ \forall x \in \mathbb{R}$. Clearly $g \ge 0$. Now let $P(x,y)$ denote the assertion $$ g(g(x)-y^2) + g(2xy) = g(x^2+y^2)$$ $\bullet P(x,x) \implies g(g(x) - x^2) = 0 \forall x \in \mathbb{R}$ $\bullet \{P(x,y),P(-x,y)\} \implies g(x) = g(-x) \forall x \in \mathbb{R}$ Now, if there does not exist any $x_{0} \neq 0 \in \mathbb{R}$ for which $g(x_0) = 0$, then clearly $g(x) = x^2 \forall x$ giving the first two of the mentioned answers. So suppose $\exists \alpha \in \mathbb{R^+}$ such that $g(\alpha)=0$. Now, $$ P(a,b) \implies g(g(a)-b^2) + g(2ab) = 0 \iff g(g(a)-b^2) = g(2ab) = 0 \forall a,b \in \mathbb{R} ; \{a^2+b^2=\alpha\}$$ So we can conclude that $g(\beta) = 0 \forall \beta \in (-\alpha , \alpha)$.Now consider the expression $P(b,y)$. We have $$g(-y^2) = g(y^2) = g(b^2+y^2)$$Now, in this expression let $y = \epsilon$ where $\epsilon \in \delta{\alpha}$ so that $(\beta)^2+(\epsilon)^2 > (\alpha)^2$. Using this we can conclude that $g(x) = g(y) \forall (x,y) \in \mathbb{R}_{\ge0}$. Using the evenness of $g$, we conclude that $g$ is indeed constant throughout. So $g \equiv 0 \implies f \equiv 0$.

Solution with Aditya Khurmi, Anshul Guha, Anushka Aggarwal, Brandon Chen, Derek Liu, Dhrubajyoti Ghosh, Ethan Zhou, Max Lu, Paul Hamrick, Robin Son, Robu Vlad, Samuel Wang: The answer is $|f(x)| = |x|$ and $f(x) = 0$, which work. To show they are the only solutions, we immediately $g(x) = f(x)^2 \ge 0$, and consider the given assertion \[ P(x,y) : g(g(x)-y^2) + g(2xy) = g(x^2+y^2). \] $P(0,0)$: $x=y=0 \implies g(g(0)) + g(0) = g(0) \implies g(g(0)) = 0$. $P(0,\sqrt{g(0)})$ gives $g(g(0)-g(0)) + g(0) = g(g(0)) = 0 \implies \boxed{g(0) = 0}$. Now $P(x,0) \implies \boxed{g(g(x)) = g(x^2)}$. Also, $P(0,x)$ implies $\boxed{g\text{ is even}}$. Claim: If $g(r) = 0$ for some $r > 0$, then $g(t) = 0$ for any $-r \le t \le r$. Proof. We may choose $x$ and $y$ such that $x^2+y^2=r$ and $2xy=t$. So $g(\text{blah}) + g(t) = g(r) = 0$ and since $g \ge 0$ this forces $g(t) = 0$. $\blacksquare$ Now, define $M = \sup \left\{ x : f(x) = 0 \right\}$, so that $g(x)=0$ for $|x|<M$ and $g(x)\ne 0$ for $|x|>M$. We do cases on $M$: If $M = \infty$, that means $g$ is identically zero. If $1 < M < \infty$, then $g(g(x)) = g(x^2)$ will give a contradiction by taking $x=M$. If $0 < M \le 1$ then consider $(x,y) = (M/2, \sqrt{M-0.001M^2})$. It is contrived to satisfy $x < M$, $y^2 < M$, $2xy < M$ but $x^2+y^2 > M$, so substituting $P(x,y)$ gives a contradiction. Finally, assume $M = 0$, meaning $g(r) = 0 \implies g = 0$. Since $P(x,x)$ gives $g(g(x)-x^2) = 0$ we get $g(x) \equiv x^2$. This finishes the proof.

Here's my solution. let $g(x) = \vert f(x) \vert$. if we find all function $g : \mathbb{R} \rightarrow \mathbb{R^+} \cup \left\{0\right\}$ which follows $g(g(x)^2 - y^2 )^2 + g(2xy)^2 = g(x^2 + y^2 )^2$, the answer is any function that satisfies $\forall _{x \in \mathbb{R}} , \vert f(x) \vert = g(x)$. So, let's find $g(x)$ ! (don't forget that $\forall_{x \in \mathbb{R}} , g(x) \geq 0$ ) Let the equation $P(x,y)$. By comparing $P(x,y)$ and $P(x,-y)$ , we can get $\forall_{x \in \mathbb{R}} , g(x) = g(-x)$ $\cdots (1)$ for two real numbers $a >b >0$, $P({\sqrt{a+b}+\sqrt{a-b} \over 2} , {\sqrt{a+b}-\sqrt{a-b} \over 2})$ tells us $g(a) \geq g(b)$, which means that g(x) is a monotone increasing function when $x \geq 0$ $\cdots (2)$. $P(x , x) : g(g(x)^2 - x^2) = 0 $ Case 1. $\exists_{a \in \mathbb{R}} , \vert g(a)^2 - a^2 \vert = t > 1$ $\Rightarrow g(t) = g(-t) = 0$ because of $(2)$ , $-t \leq x \leq t \rightarrow g(x) = 0$. lets take $1 < \alpha < t$. $\rightarrow g(\alpha) = 0$ $P(\alpha , 0) : g(\alpha^2) = 0$ $\rightarrow$ $P(\alpha^2 , 0) : g(\alpha^4) = 0$ $\rightarrow$ $\cdots$ $\rightarrow$ $g(\alpha^{2^k}) = 0$ again because of $(2)$ , $\forall_{x \in \mathbb{R}^+} , g(x) = 0$ by $(1)$ , $\forall_{x \in \mathbb{R}} , g(x) = 0$ , and we are done with Case 1. Case 2. $\forall_{x \in \mathbb{R}} , \vert g(x)^2 - x^2 \vert \leq 1$ $\Rightarrow \forall_{x \in \mathbb{R}} , x^2 -1 \leq g(x)^2 \leq x^2 +1$ Let's bound the $(LHS)$ and $(RHS)$ of $P(x , y)$. $(g(x)^2 -y^2)^2 -1 + (2xy)^2 -1 \leq (LHS) \leq (g(x)^2 -y^2)^2 +1 + (2xy)^2 +1$ $(x^2 + y^2)^2 - 1 \leq (RHS) \leq (x^2 + y^2)^2 +1$ Because $(LHS) = (RHS)$ , we get the following inequality for all real numbers $(x, y)$. $-3 \leq (g(x)^2 - y^2)^2 +(2xy)^2 - (x^2 + y^2)^2 \leq 3$ If we arrange the expression in the middle, surprisingly it is $(g(x)^2 - x^2)(g(x)^2 + x^2 -2y^2)$ ! if $g(x) \neq \vert x \vert$ , it is contradiction when you blow up $y$ to infinity. So, we get $\forall_{x \in \mathbb{R}} , g(x) = x$ , and we are done with Case 2. By Case 1. and Case 2. , $\forall_{x \in \mathbb{R}} , g(x) = 0 \; or \; \forall_{x \in \mathbb{R}} , g(x) = x$ . Finally, we get the answer $\forall_{x \in \mathbb{R}} , f(x) = 0 \; or \; \forall_{x \in \mathbb{R}} , \vert f(x) \vert = \vert x \vert $ , and we are done.

Solved with eisirrational, goodbear, Th3Numb3rThr33. The answer is $f\equiv0$ and $|f(x)|=|x|$, which clearly work. Make the obvious substitution $g(x)=f(x)^2$, so that the functional equation is $g:\mathbb R\to\mathbb R_{\ge0}$, and \[g\left(g(x)-y^2\right)+g(2xy)=g\left(x^2+y^2\right).\]Let this assertion be $P(x,y)$. Some normal FE arguments give: $P(0,0)\implies g(g(0))=0$, and $\textstyle P(0,\sqrt{g(0)})\implies\boxed{g(0)=0}$. $P(0,y)\implies g(-y^2)=g(y^2)$, so $\boxed{g\text{ even}}$. $P(x,x)\implies\boxed{g(g(x)-x^2)=0}$. Now let $g(z)=0$. Note that $P(z,z)$ gives $g(-z^2)=0$, so $g(z^2)=0$. Also for each $|a|<z$, we can find $x$ and $y$ with $x^2+y^2=z$ and $2xy=a$, and since all terms in the functional equation are positive, we will have $g(a)=g(2xy)=0$. If $g(z)=0$ for $z>1$, then $g(z)=0$ for arbitrarily large $z$, and thus $g$ is constant at $0$. If $g(z)=0$ only for $z=0$, then $g(g(x)-x^2)=0$ gives $g(x)=x^2$ for all $x$. In the final case, there is an $\varepsilon>0$ such that $g(x)=0$ for all $|x|<\varepsilon$, but $g(x)\ne0$ for all $|x|>\varepsilon$. Take $0<y^2=\varepsilon-\delta$ for arbitrarily small $\delta$, and choose $\textstyle\sqrt{\delta}<x<\min(\varepsilon,\tfrac{\varepsilon}{2y})$. Then $x^2+y^2>\varepsilon$, but $g(x^2+y^2)=0$, absurd.

Let $g(x)=f(x)^2\ge0$. $P(x,y)$ will denote the assertion $g(g(x)-y^2)+g(2xy)=g(x^2+y^2)$. $P(0,0)\Rightarrow g(g(0))=0$ $P(0,f(0))\Rightarrow g(0)=0$ $P(0.5,-x)-P(0.5,x)\Rightarrow g(x)=g(-x)$ Assume there is some nonzero number $k$ with $f(k)=0$. WLOG $k>0$. Let $x\le\sqrt k$: $P\left(x,\sqrt{k-x^2}\right)\Rightarrow g(g(x)-x^2+k)+g\left(2x\sqrt{k-x^2}\right)=0\Rightarrow g\left(2x\sqrt{k-x^2}\right)=0$ For any $x\in[0,k]$, we can set $x\mapsto\sqrt{\frac{k-\sqrt{k^2-x^2}}2}$ in this to get $g(x)=0$. By evenness of $g$, $g(x)=0$ for all $x\in[-k,k]$. If $f$ has arbitrarily large zeroes, then $k\to\infty$ so $g(x)=0$ for all $x$, hence $\boxed{f(x)=0}$ which fits. Otherwise, let $u$ be largest nonnegative number such that if $x\in(-u,u)$ then $f(x)=0$ and if $x\in(-\infty,-u)\cup(u,\infty)$ then $f(x)\ne0$. If $u=0$, then $P(x,x)\Rightarrow g(g(x)-x^2)=0\Rightarrow g(x)=x^2$, so $\boxed{f(x)=xe(x)}$ for any function $e:\mathbb R\to\{-1,1\}$. If $u>1$, then $P(x,x)$ for any $x\in(0,u)$ gives $f(-x^2)=0$, so $f(x^2)=0$. By induction, this eventually implies that $f$ has arbitrarily large zeroes. If $u<1$, then $P(u,0)\Rightarrow g(g(u))=0$, so $g(u)\in(-u,u)$. For any $x>0$, let $0<\varepsilon<\frac1{2\sqrt x}$. $P\left(\varepsilon,\sqrt x\right)\Rightarrow g(x)=g(x+\varepsilon)$, and since $g$ is all-zero on an interval and periodic, we will have $g(x)=0$, contradiction with $u<1$.

The solutions are $f(x) \equiv 0$ and all functions $f$ such that $f(x)^2=x^2$ for all real $x$, which work. Define $g(x)=f(x)^2$, and let $P(x,y)$ denote the assertion that \[g\left(g(x)-y^2\right)+g(2xy)=g\left(x^2+y^2\right).\]By $P(x,-y)$, we obtain that $g(2xy)=g(-2xy)$, so $g$ is even. Claim: If $g(a)=0$ for some positive $a$, then $g(x)=0$ for all nonnegative $x$ less than $a$. Proof: Choose $x$ and $y$ such that $x^2+y^2=a$, and let $x$ and $y$ vary such that $2xy$ reaches all possible values from $0$ to $a$. Since $g(x)$ is nonnegative by definition, $g(2xy)=0$. Let $L$ be the greatest nonnegative number such that $f(x)=0$ for all $x<L$. If $L$ doesn't exist, then $g(x) \equiv 0$. If $L$ is positive, choose arbitrarily small $\epsilon$ and plug in $P\left(\epsilon,\sqrt{L-\epsilon^2+\epsilon^{100}}\right)$ to get a contradiction. If $L=0$, then $0$ is the only root of $g(x)$. By $P(x,x)$, we have $g(g(x)-x^2)=0$, so $g(x) \equiv x^2$. Thus, $f(x) \equiv 0$ and $f(x)^2=x^2$ are the only solutions.

The answer is $f(x)\equiv 0$ or $f(x)\equiv \pm x$ where the sign depends on $x$. Clearly, they both work. Let $P(x,y)$ denote the assertion in the question. $P(x,x)$ gives $f(f(x)^2-x^2)=0$ $P(x,y)-P(x,-y)$ gives $f(t)^2=f(-t)^2$ for all $t\in \mathbb{R}$ (note $t$ corresponds to $2xy$) Let $S=\{ a| f(a)=0\}$. By above, if $t\in S, -t\in S$ If $t\in S$ and $t>0$ then for any real $u$ with $|u|<t$ there exists $x,y$ such that $2xy=u$ and $x^2+y^2=t$. Thus, $P(x,y)$ gives $f(u)=0$ so $S=[-a,a]$ for some $a\ge 0$ If $a=0$ then $f(x)^2=x^2$ and we can check any function satisfying $f(x)^2=x^2$ satisfies the problem condition. If $a>0$ then $P(\epsilon, \sqrt{a})$ gives $0+0=f(\epsilon^2+a)$, breaking the maximality of $a$, so $a$ is actually not defined, i.e. $f(x)\equiv 0$ (alternatively one can just induct)

We claim all such functions $f$ such that $|f(x)|=|x|$ and $f \equiv 0$ work and they clearly work. Let $P(x, y)$ be the assertion. $$P(y, y) \rightarrow f(f(y)-y^2)=0$$We can inductively say that $f((f(y)^2-y^2)^{2k})=0$. Now let $S=\{ z | f(z)=0\}$, now choose $x$ and $y$ such that $x^2+y^2=z$ and $P(x,y)$ gives us $f(2xy)=0$. If the elements of set $S$ doesn't have an upper bound, we can have $f \equiv 0$ for arbitrarily large values and therefore $f \equiv 0$. Now let $f(x)^2=g(x)$ and $Q(x, y)$ be the assertion for this. If $f$ is injective over $0$, we have $|f(y)|=|y|$ from $P(f(0), 0)$ and $P(y, y)$. Else, suppose $z_0$ is the largest element of $S$, now choose sufficiently small $\epsilon$ and $\delta$ and $$Q(\epsilon, \sqrt{z_0-\epsilon^2+\delta}) \rightarrow g(g(z)-z_0+\epsilon^2-\delta)+g(\epsilon\sqrt{z_0-\epsilon^2+\delta})=g(z_0+\delta)$$which means $g(z_0+\delta)=0$ which is a contradiction. $\blacksquare$

Let $P(x,y)$ denote the equivalent assertion $f(f(x)-y^2)+f(2xy)=f(x^2+y^2).$ We look for other functions except the zero, since zero works. $P(0,0)$ and $P(0,\sqrt{f(0)})$ implies $f(0)=0.$ Then $P(0,\sqrt x)$ implies $f$ is even. If $f$ is injective at zero, then $P(x,x)$ gives $f(x)=x^2$; it works. Assume $f(x_0)=0$ for some nonzero $x_0.$ Let $a^2+b^2=x_0$ then we have $f(2ab)=0$ and so $f(z)=0$ for all $z\in \{-x_0,x_0\}.$ If there is some $\psi$ such that $f(z)=0$ for all $\pm z<\psi$ and $f(z)\neq 0$ else, choose $\epsilon$ to approximate zero. Then for some small $k$ we have a contradiction by $P(\epsilon, \sqrt{\psi+k}).$

Extremely unusual solution The answer is $f \equiv 0$ and $f(x)=xh(x)$ where $h \colon \mathbb{R} \to \{-1,1\}$ is an arbitrary function, which both clearly work. Substitute $g(x)=f(x)^2$, noting that $g$ is $\mathbb{R} \to \mathbb{R}_{\geq 0}$, and the assertion $P(x,y)$ becomes $$g(g(x)-y^2)+g(2xy)=g(x^2+y^2).$$From $P(0,y)$ we obtain $g(-y^2)=g(y^2)$, hence $g$ is even, so we only really have to worry about $g(\mathbb{R}_{\geq 0})$. I claim that $g$ is (nonstrictly) increasing over $\mathbb{R}_{\geq 0}$. To establish this, I will show that for all $p>q\geq 0$ we can find $x,y \in \mathbb{R}$ with $x^2+y^2=p$ and $2xy=q$. This is equivalent to looking for real solutions to $x+y=\sqrt{p+q}$ and $xy=q/2$, which are the roots of the quadratic $$r^2-r\sqrt{p+q}+\frac{q}{2}.$$The determinant of this is $p+q-2q=p-q>0$, so we indeed have $x$ and $y$ which satisfy the desired equations. Then since the codomain of $g$ is $\mathbb{R}_{\geq 0}$, $P(x,y)$ gives us $g(p)\geq g(q)$ as desired. Now, $P(x,x)$ yields $g(g(x)-x^2)=0 \implies g(|g(x)-x^2|)=0$. If $g(0) \neq 0$, then we have $|g(x)-x^2|>0$ but $g(|g(x)-x^2|)<g(0)$, which is a contradiction. Thus $g(0)=0$. If the only root of $g$ is $0$, then this implies $g(x)=x^2$ for all $x$, which gives us the $f(x)=xh(x)$ solution. Thus suppose that $g$ has some other root. We consider two cases. Case 1: There exists a root $r$ of $g$ which is greater than $\tfrac{1}{2}$. In this case, we have $g(x)=0$ for all $x \in [-r,r] \supset [-\tfrac{1}{4},\tfrac{1}{4}]$ by the increasing condition and evenness. Then $P(r,\tfrac{1}{2})$ yields $$g\left(g(r)-\frac{1}{4}\right)+g(r)=g\left(-\frac{1}{4}\right)+g(r)=g\left(r^2+\frac{1}{4}\right)$$The middle expression is clearly $0$, so $r^2+\tfrac{1}{4}$ is a root as well. If $r>\tfrac{1}{2}$, then repeatedly applying $r \to r^2+\tfrac{1}{4}$ will generate arbitrarily large roots of $g$. This means that $g$ is zero over arbitrarily large intervals $[-t,t]$ (by evenness and the increasing condition), hence $g \equiv 0$ identically, which is one of our solutions. Case 2: Every root $r$ of $g$ is at most $\tfrac{1}{2}$. Then the condition $g(g(x)-x^2)=0$ implies that $g(x)=x^2+a(x)$, where $a \colon \mathbb{R} \to [-\tfrac{1}{2},\tfrac{1}{2}]$. Fix some massive $x$ and suppose that $g(x)=x^2+a$. Then from $P(x,\tfrac{1}{2})$, we have $$g\left(x^2-\frac{1}{4}+a\right)+x^2+a=g\left(x^2+\frac{1}{4}\right) \implies x^4+\left(\frac{1}{2}+2a\right)x^2+\left(a^2+\frac{a}{2}+\frac{1}{16}+b\right)=x^4+\frac{1}{2}x^2+\left(\frac{1}{16}+c\right),$$where $a, b, c \in [-\tfrac{1}{2},-\tfrac{1}{2}]$. Since $x$ is large but $a,b,c$ are not, it follows that $a=0$, so $g(x)=x^2$ for all sufficiently large $x$. Finally, let $t$ be massive and consider $P(t,\tfrac{x}{2t})$ for fixed $x$, which implies $$g(x)=g\left(t^2+\frac{x^2}{4t^2}\right)-g\left(t^2-\frac{x^2}{4t^2}\right).$$Since $t$ is massive, we thus have $$g(x)=\left(t^2+\frac{x^2}{4t^2}\right)^2-\left(t^2-\frac{x^2}{4t^2}\right)^2=x^2,$$which holds for all $x$, hence $g(x)=x^2$ for all $x$ as desired. These two cases cover every possibility (along with the case where $g$ has a single root at $0$), so we are done. $\blacksquare$ Remark: My (in-contest!) solution to USAMO 2022/2 used the same idea in case 2 and was directly inspired by this problem so yay for me ig

Let $g(x) = f(x)^2$ so we solve for $g$ instead in \[ g(g(x) - y^2) + g(2xy) = g(x^2 + y^2) \]from ${\mathbb R}$ to ${\mathbb R}_{\ge 0}$. Denote this assertion with $P(x, y)$. Then, by $P(x, x)$ it follows that \[ g(g(x) - x^2) = 0 \]and thus by $P(-x, x)$ \[ g(-2x^2) = g(2x^2) \]so $g$ is even. Suppose that there is some $t \ge 0$ such $g(t) = 0$. Since $g(x^2 + y^2) = 0 \implies g(2xy) = 0$, it follows that $g$ is $0$ on the interval $[-t, t]$. Let $c = \min\left(t, \frac{\sqrt{t}}{2}\right)$ Then, by $P(c, \sqrt{t})$, it follows that $f(t + c^2) = 0$. This can be repeated to get $g$ is $0$ on all $x$, and similarily $f(x) = 0$. Else, $0$ is the only root so $g(x) = x^2$ and $f(x) = \pm x$ for each $x$ individually is the solution.

YaoAOPS wrote: thus by $P(-x, x)$ \[ 2(-xy) = 2(xy) \]so $g$ is even. I think you wanted to state $g(-2x^{2}) = g(2x^{2})$.

anurag27826 wrote: YaoAOPS wrote: thus by $P(-x, x)$ \[ 2(-xy) = 2(xy) \]so $g$ is even. I think you wanted to state $g(-xy) = g(xy)$. Thanks for the catch!

Posting for storage

Nice problem! We claim that the only solutions are $f\equiv0$ and $f(x)=\pm x$ for all $x\in\mathbb{R}$, which satisfy. The only constant function that works is $f\equiv0$. Now, assume that $f(x)$ is a non-constant function. Define $g:\mathbb{R}\rightarrow\mathbb{R}_{\geq0}$ such that for all $x\in\mathbb{R}$, we have $g(x)=f(x)^2$. The original functional equation becomes $$g(g(x)-y^2)+g(2xy)=g(x^2+y^2).$$Call this assertion $p(x,y)$. We show that $g(x)=x^2$ for all $x\in\mathbb{R}$. Note that comparing $p\left(\frac{1}{2},y\right)$ and $p\left(-\frac{1}{2},-y\right)$ gives $g(x)=f(x)^2=f(-x)^2=g(-x)$. So, $g(x)$ is an even function. Moreover, $p(0,y)$ gives $g(g(0)-y^2)+g(0)=g(y^2)$. If we plug $y=\sqrt{\frac{g(0)}{2}}$ (which is valid as $g(0)\geq0$), we get $g(0)=0$. Claim. For $x_1>x_2\geq0$, we have $g(x_1)\geq g(x_2)$. If equality holds, then for each $x_1\le x\le x_2$, we have $g(x)=g(x_1)=g(x_2)$. Proof.The substitution $p\left(\frac{\sqrt{x_1+x_2}+\sqrt{x_1-x_2}}{2},\frac{\sqrt{x_1+x_2}-\sqrt{x_1-x_2}}{2}\right)$ gives $$g(x_1)=g(x_2)+g\left(g\left(\frac{\sqrt{x_1+x_2}+\sqrt{x_1-x_2}}{2}\right)-\left(\frac{\sqrt{x_1+x_2}-\sqrt{x_1-x_2}}{2}\right)^2\right)\geq g(x_2).$$We conclude that if $g(x_1)=g(x_2)$ then for all $x_2\leq x\leq x_1$, we have $g(x)=g(x_1)=g(x_2)$. We now prove injectivity at $0$. Suppose there exists $\alpha>0$ such that $g(\alpha)=0$. From our claim, we have that for every $0\leq x\leq\alpha$, we have $g(\alpha)=0$. We invoke $(\spadesuit)$ to get $g(\alpha^2)=0$. If $\alpha>1$ then for each $0\leq x\leq\alpha^{2^n}$, we have $g(x)=0$. Since $n$ can be taken arbitrarily large, we have $g(x)=0$ for all $x\geq0$ and since $g(x)$ is an even function, we get $g\equiv0$. But this contradicts the fact that $g(x)$ is a non-constant function. So, $\alpha\leq1$. Note that $p(\frac{\alpha}{2},y)$ gives $$g(y^2)+g(\alpha y)=g\left(\frac{\alpha^2}4+y^2\right)$$If we take $y=\sqrt{\alpha}$ then we get $g\left(\alpha+\frac{\alpha^2}{4}\right)=0$. Consider the sequence $\langle \alpha_n\rangle_{n\geq1}$ defined by $$\alpha_1=\alpha, \ \ \ \alpha_{n+1}=\alpha_n+\frac{\alpha_n^2}{4}$$for all $n\geq1$. This sequence is unbounded and strictly increasing. Moreover, $g(x)=0$ whenever $0\leq x\leq\alpha_n$ for each $n\geq1$. Since we can take $n$ to be arbitrarily large, we conclude that $g(x)=0$ for all $x\geq0$. But $g(x)=g(-x)$. So, $g\equiv0$. This contradicts our assumption that $g(x)$ is non-constant. It follows that there does not exist any such $\alpha$. So, injectivity at $0$ is proved. From $(\spadesuit)$, we had $g(g(x)-x^2)=0$. So, $g(x)=x^2$ for all $x\in\mathbb{R}$, as desired. $\square$

Replace $f$ with $f^2$. Then, we obtain that $f(f(x)-y^2)+f(2xy)=f(x^2+y^2),$ for all real $x,y$ and with $f$ taking nonnegative values. Taking $y \rightarrow -y$ in $(*)$ we obtain that $f$ is even. Let $f(x)=g(x^2)$ with $g: \mathbb{R}_{ \geq 0} \rightarrow \mathbb{R}_{ \geq 0}$ and so we obtain $g((g(x)-y)^2)+g(4xy)=g((x+y)^2), \,\,\, (*)$ Note that $g$ is increasing, as for all $0 \leq u \leq v$ we may find $x,y$ such that $4xy=u$ and $(x+y)^2=v$. Moreover, $g(0)=0$. Indeed, $x=y=0$ in $(*)$ implies $g(g(0)^2)=0$, and so $0 \leq g(0) \leq g(g(0)^2)=0,$ as desired. Now, we split into two cases. Case 1: $\ker g = \{0 \}$. Note that $y=x$ in $(*)$ implies that $g(x)-x=0$ for all $x$, hence $g(x)=x$ for all $x$, and so we may go back to the initial problem to conclude that all solutions are $|f(x)|=|x|$. Case 2: $\ker f \neq \{0 \}$. Since $g((g(x)-x)^2)=0$, $u \in \ker f$ implies $u^{2^n} \in \ker f$ for all $n$. Hence, if $u > 1$ belongs to $\ker f$, we conclude that $g \equiv 0$, as $u^{2^n} \rightarrow +\infty$ and $g$ is increasing. If, on the other hand, $u \leq 1$ for all such $u$, then $(g(x)-x)^2 \leq 1$ for all $x$, i.e. $x-1 \leq g(x) \leq x+1$. Thus, $(g(x)-y)^2+1+4xy+1 \geq g((g(x)-y)^2)+g(4xy)=g((x+y)^2) \geq (x+y)^2-1,$ which easily turns out to be false for $y$ large unless $g(x) \leq x$. Thus, $(g(x)-y)^2+4xy \geq g((g(x)-y)^2)+g(4xy)=g((x+y)^2) \geq (x+y)^2-1,$ which is false for $y$ small unless $g(x) \geq \sqrt{x^2-1}$. However, note that $y=0$ in $(*)$ yields $\sqrt{x^4-1} \leq g(x^2)=g(g(x)^2) \leq g(x)^2$, and so $g(x) \geq \sqrt[4]{x^4-1}$. Iterating, we obtain $g(x) \geq \sqrt[2^n]{x^{2^n}-1},$ which, by taking $n \rightarrow +\infty$, implies that $g(x) \geq x$. Therefore, $g(x)=x$ for all $x$, absurd as then $\ker f = \{0 \}$. To sum up, all solutions to the initial problem are $|f(x)|=|x|$, and the zero function.

Crazy problem, at least in the way I did it. I am not quite sure if there is a flaw (most probably not), but please let me know if there are any. The solutions are $f\equiv 0$ and $|f| \equiv |x|$. $P(0,0) \implies f(f(0)^2)^2 = 0 \implies f(f(0)^2) = 0$. $P(0,f(0)) \implies f(f(0)^2 - f(0)^2) + f(0)^2 = f(f(0)^2)^2 = 0 \implies f(0) = 0$. $P(0,y) \implies f(-y^2)^2 = f(y^2)^2$. $P(x,0) \implies f(f(x)^2)^2 = f(x^2)^2$. $P(x,f(x)) \implies f(2xf(x))^2 = f(x^2 + f(x)^2)^2$. $P(x,x) \implies f(f(x)^2 - x^2) = 0$. Now, for any $b\ge a>0$, note that we can pick $x,y$ such that $2xy = a, x^2 + y^2 = b$ by $xy = \frac{a}{2}$ and $x+y = \sqrt{a+b}$ and solving for $x$ and $y$. Then by substituting such values of $x$ and $y$ into $P(x,y)$, we get, $f(x^2+y^2)^2 \ge f(2xy)^2 \implies f(b)^2 \ge f(a)^2 \implies f(x)^2$ is increasing over $\mathbb R^+$. Now, if $f(x_0) \neq 0$ for some $x_0$. Then note that $f(x_0^2 + y^2)^2 = f(f(x_0)^2 - y^2)^2 + f(2x_0y)^2 \ge f(f(x_0)^2 - y^2)^2$. Now we can pick $y = \varepsilon$ small enough such that $\varepsilon^2 < f(x_0)^2$. Thus $f(x_0^2 + \varepsilon^2)^2 \ge f(f(x_0)^2 - \varepsilon^2)^2$. So, $x_0^2 + \varepsilon^2 \ge f(x_0)^2 - \varepsilon^2$. Now taking $\varepsilon \rightarrow 0$, we get $f(x_0)^2 \le x_0^2$. Also for all $x'$ such that $f(x') = 0$, clearly $f(x')^2 \le x'^2$. Thus $f(x)^2 \le x^2$ for all $x$. Now if there exists $\alpha > 0$ such that $f(\alpha) = 0$. Let $S = \left\{x \mid f(x) = 0\right\}$. If $S$ is unbounded above, then we are already done due to non-decreasing property of $f$. Otherwise, $S$ is bounded. Let $m = \sup S$. If $m\in S$, then $f(x) = 0$ for all $0\le x \le m$. $P(m,\varepsilon) \implies f(f(m)^2 - \varepsilon^2)^2 + f(2m\varepsilon)^2 = f(m^2 + \varepsilon^2)^2$. If $m\ge \frac{1}{4}$, picking $\varepsilon = \frac{1}{2}$ gives a contradiction to the maximality of $m$ as $m^2 + \frac{1}{4} \ge m$. Thus we must have that $m < \frac{1}{4} \implies 2m\sqrt{m} < 2m\cdot \frac{1}{2} = m$. So $f(2m\sqrt{m}) = 0$. Pick $\varepsilon = \sqrt{m}$ to get a contradiction again. Finally, $m\not\in S$. Pick $\varepsilon = \frac{1}{2}$. $P(\sqrt{m},\varepsilon) \implies f(f(\sqrt{m})^2 - \varepsilon^2) + f(2\sqrt{m}\varepsilon)^2 = f(m + \varepsilon^2)^2$. $f(\sqrt{m})^2 - \varepsilon^2 \le m - \varepsilon^2 < m \implies f(f(\sqrt{m})^2 - \varepsilon^2) = 0$. Now pick a sufficiently small $\varepsilon$ such that $2\sqrt{m}\varepsilon < m$. Therefore $f(m + \varepsilon^2)^2 = 0$, contradiction. Therefore such an $\alpha$ does not exist if $f \equiv 0$. Therefore $f(f(x)^2 - x^2) = 0 \implies f(x)^2 = x^2 \implies |f(x)| = |x|$ and we are done.

The answer is $|f(x)| = \boxed{0, |x|}$, which works. Suppose that $g(x) = f(x)^2 \ge 0$. The original statement becomes \[g(g(x)-y^2) + g(2xy) = g(x^2+y^2).\] Denote this assertion as $P(x,y)$. We will quickly establish some properties of $g$: $P(0,0)$ yields $g(g(0)) = 0$. $P(0, \sqrt{g(0)})$ yields $g(0) = g(g(0)) = 0$. $P(x,0)$ yields $g(g(x)) = g(x^2)$. $P(0,x)$ yields $g(-y^2) = g(y^2)$, or that $g$ is even. This leads us to our lemma. Lemma: Suppose that $r>0$ is a root of $g$. Then, $g(x) = 0$ for all $x \in [-r,r]$. Proof: Pick $x$ and $y$ such that $x^2+y^2=r$. Then, $P(x,y)$ yields \[g(g(x)-y^2) + g(2xy) = g(r) = 0,\] which means $g(g(x)-y^2) = g(2xy) = 0$ since $g$ is nonnegative. Furthermore, $r = x^2+y^2 \ge 2xy$ due to AM-GM, covering all $|x| \le r$. $\square$ Plugging in $P(r,r)$ yields \[g(g(r)-r^2) = g(-r^2) = g(r^2) = 0.\] If $r>1$, then $r^2>r$, implying that there is no upper bound within the roots of $g$. Then, the lemma shows $g$ is constant. Now, suppose that $0<r\le 1$. Then, $P(\tfrac{r}{2}, \sqrt{r})$ yields \[g\left(g\left(\frac{r}{2} \right) - r\right) + g(r \sqrt{r}) = g\left(\frac{r^2}{4}+r \right).\] However, the lemma gives that $g(\tfrac{r}{2})=0$, so the first term is equal to $0$. Then, $r \sqrt{r} \le r$, so the second term equals $0$ as well. Hence, $g(\tfrac{r^2}{4}+r) = 0$. Since $r>0$, the upper bound within the roots of $g$ is still unbounded, which implies that $g$ is constant by the lemma. That means if there exists such a value of $r$, then $g \equiv 0$. If not, then $0$ is the only root of $g$, which means that $g(x) = 0$ implies $x=0$. Finally, $P(x,x)$ yields \[g(g(x)-x^2) = 0 \implies g(x)=x^2.\] Since $g(x) = f(x)^2$, we see that $f(x) = \pm x$ and checking for piecewise function yields $f(x) = \pm x$. Condensing this into $|f(x)| = |x|$ gives us our second solution set.