Clearly $f$ is injective and a well known result states that a continuous and injective function is monotone. Suppose $f$ is increasing, if $f(x)>x$ then $f(f(x))\ge f(x) > x$ and $f(f(f(x)))\ge f(f(x))>x$ so their sum is greater than $3x$ which is a contradiction therefore $x \ge f(x)$, meaning $x \ge f(x)\ge f(f(x))$ and $x\ge f(x) \ge f(f(x)) \ge f(f(f(x)))$ so their sum is $\le 3x$ and equality must hold everywhere giving $f(x)=x$. The case when $f$ is decreasing is not possible because $x>0 \implies f(x) < f(0) = 0$. @below fixed

MrOreoJuice wrote: The case when $f$ is decreasing is similar. No, since then $f(f(x))$ and $f(x)$are not monotonous in the same direction.

In fact, we need neither the condition $f(0)=0$ nor the continuity: Fix $x$ and consider the sequence $a_n=f^{(n)}(x)$ so that $a_{n+3}+a_{n+2}+a_{n+1}=3a_n$. But from the explicit formula of such a recurrence sequence it is easy to see that such a sequence is either constant or negative at some point. Since the latter is excluded, we find that $a_n$ is constant and in particular $f(x)=a_1=a_0=x$. Since $x$ was arbitrary, the claim follows.

Same idea as this P3: https://www4352.vu.lt/matematikos-olimpiados/wp-content/uploads/2015/11/mifmo2015.pdf

R8kt wrote: Same idea as this P3: https://www4352.vu.lt/matematikos-olimpiados/wp-content/uploads/2015/11/mifmo2015.pdf Well, of course the trick is much older (for instance see this problem from the IMO Shortlist 1992).

R8kt wrote: Same idea as this P3: here