The following result is used in my solution . Let $ a,b$ is two real number then the function $ ax+b$ doesn't change on $ R$ if and only if $ a=0$ Consider two case : 1. $ a>0$ Take $ x=0$ then $ af(y)\leq y+f(f(0))$ Take $ y=0$ then $ x+af(0)\leq f(f(x))$ Then take $ y\to f(y)$ we have \[ af(f(y))\leq f(y)+f(f(0)) \] Because $ a>0$ so $ ay+a^2f(0)\leq af(f(x))\leq f(y)+f(f(0))$ .Use the first inequality we have $ a^2y+a^3f(0)\leq af(y)+af(f(0))\leq y+(a+1)f(f(0))$ , and by above result we obtain that $ a^2=1$ thus $ a=1$ In this case we can find out that $ f(x)=x$ satisfy condition . Case 2 :$ a=0$ ,it is easy to check that it doesn't satisfy condition Case 3 : Let consider $ f(x)=x^2+k$ where $ k>0$ .We wish exist k such that : $ x+a(y^2+k)\leq y+(x^2+k)^2+k$ Only need to choose $ k$ is large enough then $ (x^2+k)^2+k-x\geq 2kx^2-x+k^2+k>0$ and $ -a(y^2+k)+y>0$ Thus all a satisfy condition are $ a=1$ or $ a<0$

you are even allowed to use that the function $ a_nx^n + a_{n-1}x^{n-1}+\dots +a_1x + a_0$ doesn't change on the whole $ \mathbb{R}$ iff $ a_n = a_{n - 1} = \dots = a_1 = 0$ your solution is pretty elegant, by the way, congratulations.

I got a very long solution that I don't want to write, for the part that TTsphn did. Yet, as I can see the examples aren't written. For $ a=1$, $ f(x)=x$; for $ -1<a<0$, $ f(x)=\mid\frac{x}{a}\mid$; for $ -1\geq a$, $ f(x)=\mid x\mid$

Consider the following substitutions in the original inequality: $ x = y$: $ af(x) \le f(f(x))$. $ y = f(x)$: $ x + af(f(x)) \le f(x) + f(f(x)) \leftrightarrow f(x) - x \ge (a - 1)f(f(x)) \ge (a - 1)af(x) \leftrightarrow (1 + a - a^2)f(x)\ge x$ It's clear that when $ a + 1 - a^2$ is nul, we'll have of any real $ x$ that $ 0 \ge x$, contradiction. So we have two case to look out: $ 1 + a - a^2 > 0 \rightarrow f(x) \ge \dfrac{x}{1 + a - a^2}$. We can get: $ y + f(f(x)) \ge x + af(y) \ge x + a\dfrac{y}{1 + a - a^2} \leftrightarrow f(f(x)) - x \ge y(\dfrac{a}{1 + a - a^2} - 1)$. When we fix $ x$, if $ \dfrac{a}{1 + a - a^2} - 1\not = 0$, we can variate $ y$ such that the expression $ y(\dfrac{a}{1 + a - a^2} - 1)$ gets as big as we want, absurd because $ f(f(x)) - x$ is greater than theis expression. So this expression is nul, giving us the possible values $ 1$ and $ - 1$ for $ a$ (it's easy to find such a function to both). $ 1 + a - a^2 < 0 \rightarrow f(x) \le \dfrac{x}{1 + a - a^2}$. Then: $ x + af(y) \le y + f(f(x)) \le y + \dfrac{f(x)}{1 + a - a^2} \le y + \dfrac{x}{(1 + a - a^2)^2}$, which implies: $ af(y) - y \le x(\dfrac{1}{(1 + a - a^2)^2} - 1)$. By a similar argument, the right expression is nul, and leave us just to see the existence of solutions to $ a = 2$. Some work after I think shows that there are no such functions to $ a = 2$

When you set $ y = f(x)$, you get $ f(x) - x\ge (a - 1)f(f(x))\ge (a - 1)af(x)$. But the last inequality holds only when $ a\ge1$.

When you mention "the last inequality", do you want to mention $ (a-1)f(f(x)) \ge a(a-1)f(x)$? If you mention it, it's true because I had already proved that $ af(x) \le f(f(x))$. If it's not, would you explain me better?

If $ a<1$, $ af(x)\le f(f(x))\Rightarrow (a-1)f(f(x))\le a(a-1)f(x)$, not $ (a-1)f(f(x))\ge a(a-1)f(x)$. In other words, your argument only works for the case $ a\ge1$ given the other part is all correct.

Now I see how wrong I was! I always get trouble with signals, maybe because sometimes I don't pay attention to them. Well, it means that it's missing in the solution the case when $ a<1$.

it's trivial that $a\ne 0$;if $a<0$,then let $f(x)=c|x|$ where c is sufficiently large. if $a>0$,then fix $x$(or y),we get that f(x) can be sufficiently large(both positive and negative). let $x=y$ then $f(f(x))>af(x)$. and by letting $y=f(x)$,we can get $f(f(x))\le\frac{f(x)}{a}+d$(where $d=\frac{f(f(t))-t}{a} $ is fixede) if $a>1$,choose an $x$ for which $f(x)$ is sufficiently large,then $af(x)>\frac{f(x)}{a}+d$,contradiction. if $a<1$ let $|f(x)|$ be sufficiently large an analagous contradiction. hence $a=1$ or $a<0$.

TTsphn wrote: The following result is used in my solution . Let $ a,b$ is two real number then the function $ ax+b$ doesn't change on $ R$ if and only if $ a=0$ Consider two case : 1. $ a>0$ Take $ x=0$ then $ af(y)\leq y+f(f(0))$ Take $ y=0$ then $ x+af(0)\leq f(f(x))$ Then take $ y\to f(y)$ we have \[ af(f(y))\leq f(y)+f(f(0)) \]Because $ a>0$ so $ ay+a^2f(0)\leq af(f(x))\leq f(y)+f(f(0))$ .Use the first inequality we have $ a^2y+a^3f(0)\leq af(y)+af(f(0))\leq y+(a+1)f(f(0))$ , and by above result we obtain that $ a^2=1$ thus $ a=1$ In this case we can find out that $ f(x)=x$ satisfy condition . Case 2 :$ a=0$ ,it is easy to check that it doesn't satisfy condition Case 3 : Let consider $ f(x)=x^2+k$ where $ k>0$ .We wish exist k such that : $ x+a(y^2+k)\leq y+(x^2+k)^2+k$ Only need to choose $ k$ is large enough then $ (x^2+k)^2+k-x\geq 2kx^2-x+k^2+k>0$ and $ -a(y^2+k)+y>0$ Thus all a satisfy condition are $ a=1$ or $ a<0$ by above result?

a=1 and a- negative

Let $c = f(f(0))$ for brevity. When $a = 1$, clearly $f(x) = x$ works, so we consider three general cases: Case 1. $a>1$. We'll show no suitable $f$ exists. Assume the contrary and plug in $x=y$ to obtain $f(f(x)) \geq af(x)$ $\forall x\in\mathbb{R}$. Then substituting $y = f(z)$ and $x = 0$, we get $af(z) \leq af(f(z)) \leq f(z) + c \Longrightarrow f(z) \leq \frac{c}{a-1}$ $\forall z\in\mathbb{R}$. Now that we've shown that the image of $f$ is bounded from above, we can pick $x > \frac{c}{a-1} - af(0)$ and $y=0$ to derive the contradiction: \[x + af(0) \leq f(f(x)) \leq \frac{c}{a-1} < x + af(0).\]Case 2. $a \in (0,1)$. There are no solutions in this case either. Assume the contrary and plug in $x = 0$ to get $af(y) \leq y + c \Longrightarrow f(y) \leq \frac{y+c}{a}$ $\forall y\in\mathbb{R}$. Now substituting $y = 0$ and some $x < \frac{-c-ca+a^3f(0)}{1-a^2}$, we get a contradiction: \[x + af(0) \leq f(f(x)) \leq \frac{f(x) + c}{a} \leq \frac{\left(\frac{x+c}{a}\right)+c}{a} = \frac{x}{a^2} + \frac{c}{a^2} + \frac{c}{a} \Longrightarrow x \geq \frac{-c-ca+a^3f(0)}{1-a^2}.\] Case 3. $a < 0$. Here we can actually construct functions satisfying the inequality. For $a \leq -1$, taking $f(x) = |x|$ works because $x \leq |x|$ and $a|y| \leq -|y| \leq y$. For $a \in (-1, 0)$, taking $f(x) = \frac{|x|}{a^2}$ works because $x \leq |x| \leq \frac{|x|}{a^4}$ and $\frac{|y|}{a} \leq -|y| \leq y$, so we're done! Finally, the answer is $\boxed{a \in (-\infty, 0) \cup \{1\}}$.