Suppose some k so that f(k) = k. Now k = ak + bk, so that a+b = 1, contradiction. --- Case 1: f(x) - x > 0 for all x. Then f(f(x)) > f(x), whereby (a-1)f(x) + bx > 0 for all x. We have bx < b f(x), whereby (a+b)f(x) > f(x), or a+b > 1, contradiction. --- Case 2: f(x) - x < 0 for all x. Then f(f(x)) < f(x), where again by a similar process to Case 1, we get a+b > 1, contradiction. ==== Thus, there is no such f. *note: i didnt check my proof but it appears right.

Your proof is wrong. Firstly you can have $f(0)=0$ so you cannot assume that for all $x$ you have $f(x)>x$. Secondly there exists at least two such functions: $f(x)=cx$ where $c$ is the root of $x^2-ax-b=0$ But the good thing about your post above is that it can be used to prove $f(0)=0$.

This has been incredibly frustrating. This problem was used in a very old Romanian NMO, I distinctly remembered solving a very long time ago, but couldn't nail it . Let $t>0,t'<0$ be the roots of $x^2-ax-b=0$, put $u=\frac 1t,u'=\frac 1{t'}$, and let $g=f^{-1}$ ($f$ is continuous and injective, and thus bijective, and also strictly monotonic). Notice that $|t|>|t'|\Rightarrow|u|<|u'|\ (*)$. I'll use the fact that $f(0)=0$, which can indeed be shown by employing the method used by singular. We now write the recurrences for both $f$ and $g$. We get $f^{(n)}(x)=\frac{f(x)-t'x}{t-t'}\cdot t^n+\frac{tx-f(x)}{t-t'}\cdot t'^n\ (1)$ and $g^{(n)}(x)=\frac{g(x)-u'x}{u-u'}\cdot u^n+\frac{ux-g(x)}{u-u'}\cdot u'^n\ (2)$. Now, $(*),(1),(2)$ imply that for a fixed $x$ and large $n,\ f^{(n)}(x)$ will have the same sign as $\frac{f(x)-t'x}{t-t'}\cdot t^n\ (\#)$, while $g^{(n)}(x)$ will have the same sign as $\frac{ux-g(x)}{u-u'}\cdot u'^n\ (\#\#)$ (unless $f(x)=t'x$ and $g(x)=ux$, respectively). Suppose first that $f$ is strictly increasing. In this case, we will have $f(x)>0,\ \forall x>0$, which, in turn, means that $g^{(n)}(x)>0,\ \forall x>0,\ \forall n$. If we could find some $x$ for which $g(x)\ne ux$, then for large $n,\ (\#\#)$ would change sign infinitely often, and so would $g^{(n)}(x)$, according to the last observation in the previous paragraph, thus yielding a contradiction. It means that in this case $g(x)=ux\iff f(x)=tx$. Assume now that $f$ is strictly decreasing. In this case $f^{(n)}(x)$ has alternate signs as $n$ increases (for every $x$). If we could find an $x$ for which $f(x)\ne t'x$, for large enough $n,\ (\#)$ would have constant sign, and then so would $f^{(n)}(x)$, leading to contradiction. This means that when $f$ is decreasing we have $f(x)=t'x$. I hope it's correct. It would be very dissapointing if it wasn't .

I'm not sure if your solution is correct or not, though I can't find any mistakes... I must note that none of the 19 students writing this problem got it right, though.

I thing that this kind of functional equation could be solved by using this sequence: $x_{0}=x$ $f(x_{n})=x_{n+1}$ Reporting this in the original equation gives: $x_{n+2}=ax_{n+1}+bx_{n}$ And thus there exist two reals $A$ and $B$ such that: $x_{n}=A(\frac{a+\sqrt{a^2+4b}}{2})^n+B(\frac{a-\sqrt{a^2+4b}}{2})^n$ Now ( The tricky part) by using the propreties of $f$ we can find an argument to eliminate alternatively $A$ or $B$. And conclude by putting$x_{1}=f(x_{0})$

Nothing new here . That's exactly what I did in order to get $(1)$ and $(2)$.

Thank you very much.

grobber wrote: This has been incredibly frustrating. This problem was used in a very old Romanian NMO, I distinctly remembered solving a very long time ago, but couldn't nail it . In fact, the problem proposed at the Romanian NMO (author M. Piticari) only asked to prove that $f(0)=0$