Define $g : \mathbb{R} \setminus \{-1\} \to \mathbb{R} \setminus \{-1\}$ by $g(y) = \frac{1 - y}{1 + y} $ For any $x, y \in \mathbb{R}$ such that $(x + 1)(y + 1)(xy + 1) \ne 0$, Let $P(x, y):~f(x)f \left( f (g(y)\right) = f\left(\frac{x + y}{xy + 1}\right) $. Since the image of $g$ is $\mathbb{R} \setminus \{-1\}$, we can deduce that $\mathrm{Im}(f) \subset \mathbb{R} \setminus \{-1\}$. $P(x, 0)$ gives $f(x)(f(f(1)) - 1)=0$ for all $x \in \mathbb{R} \setminus \{-1\}$. - Case 1: $f(f(1)) \neq 1$. In this case $f=0$ which is indeed a solution. - Case 2: $f(f(1)) = 1$. $P(x,0)$ gives $f(x)f(f(0))=f(1)$ for all $x \in \mathbb{R} \setminus \{-1\}$. - Case 2.1: $f(f(0)) \neq 0$ In this case, $f$ is constant, since $f(f(1))=1$, then $f=1$ which is indeed a solution. - Case 2.2: $f(f(0)) = 0$ In this case, $f(1)=0$ which implies $f(0)=1$. $P(0,x)$ gives $Q(x)$: $f(f(g(x)))=f(x)$ for all $x \in \mathbb{R} \setminus \{-1\}$. Therefore $P$ becomes: $$f(x)f(y)=f\left(\frac{x + y}{xy + 1}\right)$$for any $x, y \in \mathbb{R}$ such that $(x + 1)(y + 1)(xy + 1) \ne 0$. for any $x \in \mathbb{R} \setminus \{-1, 1\}$, $P(x, -x)$ gives $f(x)f(-x)=1$, which we denote $R(x)$. - Case 2.2.1: There exist $t \in \mathbb{R} \setminus \{-1, 0, 1\}$ such that $f(t)=1$. $P(x, t)$: $f(x)=f\left(\frac{x + t}{xt + 1}\right)$ for all $x \in \mathbb{R} \setminus \{-1, \frac{-1}{t}\}$ $P(\frac{-1}{t}, \frac{x + t}{xt + 1})$: gives $f(x)f(\frac{-1}{t})=f(\frac{1}{x})$ for all $x \in \mathbb{R} \setminus \{-1, 0, \frac{-1}{t}\}$ Take $x=t$ in the last equation, and use the property $R$ to get $f(\frac{1}{t})=f(\frac{-1}{t})=1$. Therefore, $f(x)=f(\frac{1}{x})$ for all $x \in \mathbb{R} \setminus \{-1, 0, \frac{-1}{t}\}$, since this property is also true for $\frac{-1}{t}$, then it is true for all $x \in \mathbb{R} \setminus \{-1, 0 \}$. Now, $Q(g(x))$: give $f(f(x))=f(g(x))$ for all $x \neq -1$. $Q(g(\frac{1}{x}))$ gives $f(f(x))=f(f(\frac{1}{x}))=f(g(1/x))=f(-g(x))=\frac{1}{f(g(x))}$ (we used the property $R$ again) for all $x \in \mathbb{R} \setminus \{-1, 0 \}$. Therefore $f(g(x))=1$ for all $x \in \mathbb{R} \setminus \{-1, 0 \}$ which is equivalent to $f(x)=1$ for all $x \in \mathbb{R} \setminus \{-1, 1 \}$. This yields to the function $f(x)=\begin{cases} 0 &\mbox{if } x=1\\ 1 & \mbox{if } x \in \mathbb{R} \setminus \{-1, 1 \} \end{cases}$ which is not a solution. - Case 2.2.2: $f(t)=1 \implies t=0$ Let $x, y \in \mathbb{R} \setminus \{-1\} $ such that $f(x)=f(y)=z$. If $x=1$, then $f(y)=0$. Hence if $y \neq 1$ then $R(y)$ gives $0=f(y)f(-y)=1$ which is absurd. Therefore $y=x=1$. We treat $y=1$ in the same way. Now suppose $x, y \in \mathbb{R} \setminus \{-1, 1\} $, looking back at the property $R$, we see that $z \neq 0$. If $xy \neq 1$. $P(x, -y)$ gives $f(x)f(-y)=f(\frac{x-y}{xy-1})$, $R(y)$ gives $f(-y)=\frac{1}{z}$, therefore $f(\frac{x-y}{xy-1})=1$ which implies $x=y$. If $xy=1$. $Q(g(x))$ and $Q(g(\frac{1}{x}))$ give $f(g(x))=f(f(x))=f(f(\frac{1}{x}))=f(-g(x))=\frac{1}{f(g(x))}$, which imples $f(g(x))=1$ $\implies$ $g(x)=0$ $\implies$ $x=1$. We deduce that $f$ is injective. $Q(g(x))$ gives $f(f(x))=f(g(x))$ $\implies$ $f(x)=g(x)$ for all $x \in \mathbb{R} \setminus \{-1\}$, which is indeed a solution. Conclusion The solutions are: $f(x)=0$ $\forall x \in \mathbb{R} \setminus \{-1\}$ $f(x)=1$ $\forall x \in \mathbb{R} \setminus \{-1\}$ $f=g$

cazanova19921 wrote: Define $g : \mathbb{R} \setminus \{-1\} \to \mathbb{R} \setminus \{-1\}$ by $g(y) = \frac{1 - y}{1 + y} $ For any $x, y \in \mathbb{R}$ such that $(x + 1)(y + 1)(xy + 1) \ne 0$, Let $P(x, y):~f(x)f \left( f (g(y)\right) = f\left(\frac{x + y}{xy + 1}\right) $. Since the image of $g$ is $\mathbb{R} \setminus \{-1\}$, we can deduce that $\mathrm{Im}(f) \subset \mathbb{R} \setminus \{-1\}$. $P(x, 0)$ gives $f(x)(f(f(1)) - 1)=0$ for all $x \in \mathbb{R} \setminus \{-1\}$. - Case 1: $f(f(1)) \neq 1$. In this case $f=0$ which is indeed a solution. - Case 2: $f(f(1)) = 1$. $P(x,0)$ gives $f(x)f(f(0))=f(1)$ for all $x \in \mathbb{R} \setminus \{-1\}$. - Case 2.1: $f(f(0)) \neq 0$ In this case, $f$ is constant, since $f(f(1))=1$, then $f=1$ which is indeed a solution. - Case 2.2: $f(f(0)) = 0$ In this case, $f(1)=0$ which implies $f(0)=1$. $P(0,x)$ gives $Q(x)$: $f(f(g(x)))=f(x)$ for all $x \in \mathbb{R} \setminus \{-1\}$. Therefore $P$ becomes: $$f(x)f(y)=f\left(\frac{x + y}{xy + 1}\right)$$for any $x, y \in \mathbb{R}$ such that $(x + 1)(y + 1)(xy + 1) \ne 0$. for any $x \in \mathbb{R} \setminus \{-1, 1\}$, $P(x, -x)$ gives $f(x)f(-x)=1$, which we denote $R(x)$. - Case 2.2.1: There exist $t \in \mathbb{R} \setminus \{-1, 0, 1\}$ such that $f(t)=1$. $P(x, t)$: $f(x)=f\left(\frac{x + t}{xt + 1}\right)$ for all $x \in \mathbb{R} \setminus \{-1, \frac{-1}{t}\}$ $P(\frac{-1}{t}, \frac{x + t}{xt + 1})$: gives $f(x)f(\frac{-1}{t})=f(\frac{1}{x})$ for all $x \in \mathbb{R} \setminus \{-1, 0, \frac{-1}{t}\}$ Take $x=t$ in the last equation, and use the property $R$ to get $f(\frac{1}{t})=f(\frac{-1}{t})=1$. Therefore, $f(x)=f(\frac{1}{x})$ for all $x \in \mathbb{R} \setminus \{-1, 0, \frac{-1}{t}\}$, since this property is also true for $\frac{-1}{t}$, then it is true for all $x \in \mathbb{R} \setminus \{-1, 0 \}$. Now, $Q(g(x))$: give $f(f(x))=f(g(x))$ for all $x \neq -1$. $Q(g(\frac{1}{x}))$ gives $f(f(x))=f(f(\frac{1}{x}))=f(g(1/x))=f(-g(x))=\frac{1}{f(g(x))}$ (we used the property $R$ again) for all $x \in \mathbb{R} \setminus \{-1, 0 \}$. Therefore $f(g(x))=1$ for all $x \in \mathbb{R} \setminus \{-1, 0 \}$ which is equivalent to $f(x)=1$ for all $x \in \mathbb{R} \setminus \{-1, 1 \}$. This yields to the function $f(x)=\begin{cases} 0 &\mbox{if } x=1\\ 1 & \mbox{if } x \in \mathbb{R} \setminus \{-1, 1 \} \end{cases}$ which is indeed a solution. - Case 2.2.2: $f(t)=1 \implies t=0$ Let $x, y \in \mathbb{R} \setminus \{-1\} $ such that $f(x)=f(y)=z$. If $x=1$, then $f(y)=0$. Hence if $y \neq 1$ then $R(y)$ gives $0=f(y)f(-y)=1$ which is absurd. Therefore $y=x=1$. We treat $y=1$ in the same way. Now suppose $x, y \in \mathbb{R} \setminus \{-1, 1\} $, looking back at the property $R$, we see that $z \neq 0$. If $xy \neq 1$. $P(x, -y)$ gives $f(x)f(-y)=f(\frac{x-y}{xy-1})$, $R(y)$ gives $f(-y)=\frac{1}{z}$, therefore $f(\frac{x-y}{xy-1})=1$ which implies $x=y$. If $xy=1$. $Q(g(x))$ and $Q(g(\frac{1}{x}))$ give $f(g(x))=f(f(x))=f(f(\frac{1}{x})=f(-g(x))=\frac{1}{f(g(x)}$, which imples $f(g(x))=1$ $\implies$ $g(x)=0$ $\implies$ $x=1$. We deduce that $f$ is injective. $Q(g(x))$ gives $f(f(x))=f(g(x))$ $\implies$ $f(x)=g(x)$ for all $x \in \mathbb{R} \setminus \{-1\}$, which is indeed a solution. Conclusion The solutions are: $f(x)=0$ $\forall x \in \mathbb{R} \setminus \{-1\}$ $f(x)=1$ $\forall x \in \mathbb{R} \setminus \{-1\}$ $f(x)=\begin{cases} 0 &\mbox{if } x=1\\ 1 & \mbox{if } x \in \mathbb{R} \setminus \{-1, 1 \} \end{cases}$ $f=g$ The case $f(x)=\begin{cases} 0 &\mbox{if } x=1\\ 1 & \mbox{if } x \in \mathbb{R} \setminus \{-1, 1 \} \end{cases}$ seems wrong. If we take $x=0$, $y=2$ in the original equation, we have $$f(0)f \left( f \left(\frac{1 - 2}{1 + 2} \right)\right) = f(2) $$So $$0=1\times 0=1\times f(1)=1$$

Thanks for your observation.