If $h=f \circ g^{-1}$, then $h^{-1}=g \circ f^{-1}$, so $h(x)+h^{-1}(x)=2x$. Let $d_{x}=h(x)-x$. It follows that \[ h^n(x)=x+nd_{x}, \forall n \in {\mathbb {Z}}. \] For $n<0$, $h^n$ means $(h^{-1})^{-n}$. $h$ is continuous and bijective, hence it is strictly increasing (since it cannot be decreasing). Let $x<y$. Because $h$ is strictly increasing, \[ h^n(x)<h^n(y) \Rightarrow x+nd_{x}<y+nd_{y} \Rightarrow (y-x)+n(d_{y}-d_{x}) > 0, \forall n \in {\mathbb {Z}}, \] hence $d_{x}=d_{y}$, or $d_{x}$ constant, so $h(x)=x+c$. Since $h(x_{0})=x_{0}$, $c=0$ and $h=1_{\mathbb {R}}$, i.e, $f=g$.

Thank you! The book had a long and akward solution. . To clarify one thing. You took $x_{0}=f(x_{0})=g(x_{0})$ to obtain $h(x_{0})=x_{0}$, right?

a mistake...it should say $h(g(x_0)) = g(x_0)$ rather than $h(x_0) = x_0$. the remainder of the proof isn't affected, of course

Ohh.. I didn't understood either why $d_{x}=d_{y}$ Can you please explain a bit, if you don't mind? My guessing: $(y-x)$ is positive and to have LHS positive then $m=d_{y}-d_{x}< 0$.. or $\leq 0$? but $m$ is always positive?

$(y-x)+n(d_{y}-d_{x}) > 0, \forall n \in {\mathbb {Z}}$ if $d_{y}-d_{x} < 0$, then as $n$ goes to $\infty$, $n(d_{y}-d_{x})$ goes to $-\infty$, and if $d_{y}-d_{x} > 0$, then as $n$ goes to $-\infty$, $n(d_{y}-d_{x})$ goes to $-\infty$. both of these contradict the statement above. p.s. esti roman? si eu

Another question (after some time) Why $h^n(x)=x+nd_{x}, \forall n \in {\mathbb {Z}}.$ Could this problem be generalised?

$h(x) + h^{-1}(x) = 2x$ (**) and $d_x = h(x) - x$ (by definition), so $h(x) = x + d_x$ and $h^{-1}(x) = 2x - h(x) = 2x - (h_x + x) = x - d_x$. then induce: assume $h^n (x) = x+nd_x$ is true for all $n \in [-k, k]$, $k\in \mathbb N$. from (**) we get (by replacing $x$ with $h^k (x)$) $h^{k+1} (x) + h^{k-1} (x) = 2h^k (x) = 2x+2kd_x$ and since $k-1$ is in the range $[-k, k]$, this becomes $h^{k+1}(x) + x+(k-1)d_x = 2x+2kd_x$; and then $h^{k+1}(x) = x + (k+1)d_x$. a similar argument (replacing $x$ with $h^{-k} (x)$ in (**)) gives that $h^{-k-1}(x) = x + (-k-1)d_x$. so the relation holds on $[-k-1, k+1]$--induction complete.