A solution from Kalva: Suppose f(a) = a. Then putting x = y = a in the relation given, we get f(b) = b, where b = 2a + a2. If -1 < a < 0, then -1 < b < a. But f(a)/a = f(b)/b. Contradiction. Similarly, if a > 0, then b > a, but f(a)/a = f(b)/b. Contradiction. So we must have a = 0. But putting x = y in the relation given we get f(k) = k for k = x + f(x) + xf(x). Hence for any x we have x + f(x) + xf(x) = 0 and hence f(x) = -x/(x+1). Finally, it is straightforward to check that f(x) = -x(x+1) satisfies the two conditions. Thanks to Gerhard Woeginger for pointing out the error in the original solution and supplying this solution.

Or rather (a typo in the last line of Arne's solution) \[f(x)=\frac{-x}{x+1}.\]

Another (trace of) solution: - find f(0)=0 - now get $ f(f(x)) = x$ - substitute f(y) for y and get $ f(x + y + xy) = f(x) + f(y) + f(x)f(y)$ - define $ g(x) = f(x - 1) + 1$ (we conjugate f) and get the equivalent problem: - g is from the positive reals to themselves - $ g(ab) = g(a)g(b)$ - $ g(g(x)) = x$ - and $ \frac {g(x) - g(1)}{x - 1}$ is increasing these conditions are no doubt very nice And from now on the problem is standard... for example, you can fix $ f(2)$ and then extend uniquely f to a dense set of points, use the last condition to get continuity, and use $ g^2=\mbox{id}$ to get $ g(x) = 1/x$ for all x.

$\frac{f(x)}{x}$ is strictly increasing on $(-1,0)$ and $(0,\infty) \Rightarrow f(x)=x$ can have atmost three solutions(atmost one in the interval $(-1,0)$,another probably in the interval $(0,\infty)$ and the third one probably $0$).Let $k$ be a fixed point of $f$ in the interval $(-1,0)$.Then $x=y=k \Rightarrow f(k^2+2k)=k^2+2k \Rightarrow k^2+2k$ is also a fixed point of $f$.Moreover $k \in (-1,0) \Rightarrow -1 \le k(k+2) <0$ forcing $2k+k^2=k \Rightarrow k=0.-1$.Contradiction!!Similarly it may be seen that there are no fixed points of $f$ in the interval $(0,\infty)$.But substituting $y=x$ in the equation we get $f(x+f(x)+xf(x))=x+f(x)+xf(x)$ and hence we are forced to conclude that $x+f(x)+xf(x)=0 \Rightarrow f(x)=-\frac{x}{x+1}$It is easy to see that this indeed satisfies the relation.

IF $ f(x + f(y) + xf(y)) \geq y + f(x) + yf(x)$，and then how to solve it?

anyone solve it...

any one look?

The function $f(x)$ can have at most 3 fixed points, one in the interval $(-1,0)$, one at $0$, and one in the interval $(0,\infty)$. Now plug in $y=x$ into the FE. We get \[ f(x+f(x)+xf(x)) = x+f(x)+xf(x).\]This means that $x+f(x)+xf(x)$ is also a fixed point. Therefore if $x$ is a fixed point, so is $2x+x^2$. Note that $2x+x^2$ is always positive for positive $x$, so we must have $x \in (-1,0)$ and $2x+x^2 \in (0,\infty)$. However, when $x \in (-1,0)$, $2x+x^2<0$. This means that we must have a fixed point at $0$. Therefore, since $x+f(x)+xf(x)$ is a fixed point, we must have \[ x+f(x)+xf(x) = 0\implies f(x) = -\frac{x}{x+1}.\]It is easy to check that this function satisfies the conditions. EDIT: 1000th post!

Here is my solution for this problem Solution $f(x + f(y) + xf(y)) = y + f(x) + yf(x)$ Let $y = x$, we have: $f(x + f(x) + xf(x)) = x + f(x) + xf(x)$ So: $x + f(x) + xf(x)$ is fixed point of $f$ Suppose $u \in S$ is fixed point of $f$, which means $f(u) = u$ Let $x = y = u$, we have: $f(u^2 + 2u) = u^2 + 2u$ If $u \ne u^2 + 2u$ then $1 = \dfrac{f(u)}{u} \ne \dfrac{f(u^2 + 2u)}{u^2 + 2u} = 1$ So: $u = u^2 + 2u$ or $u = 0, u = - 1$ It means that there is no fixed point in $(- 1; 0) \cup (0; + \infty)$ Let $x = y = 0$, we have: $f(f(0)) = f(0)$ Then: $f(0)$ is fixed point of $f$ So: $f(0) = 0$ or $x + f(x) + xf(x) = 0$ Hence: $f(x) = - \dfrac{x}{x + 1}, \forall x \in S$ Retry, we see that: $f(x) = - \dfrac{x}{x + 1}$ satisfies the problem In conclusion, we have: $f(x) = - \dfrac{x}{x + 1}, \forall x \in S$

$P(0,y) \implies f(f(y))=y+f(0)+yf(0)$ thus $f$ is injective put$ y=0 $then$ ff(0))=f(0) \implies f(0)=0$ then $ff(x))=x \implies \frac{ff(x))}{f(x)}=\frac{x}{f(x)}$ if $x>0$ and $f(x)>x$ then $\frac{ff(x))}{f(x)}=\frac{x}{f(x)}>\frac{f(x)}{x} $then $f(x)<x$ contradiction thus if $x>0$ then $f(x)<0$ and vice versa $P(x,x) \implies f(x+f(x)+xf(x))=x+f(x)+xf(x) \implies x+f(x)+xf(x)=0 $ then $f(x)=\frac{-x}{x+1} $ $\forall x\in S$

Let $P(x,y)$ be the assertion. $P(x,x)\implies f(x+(1+x)f(x))=x+(1+x)f(x). ~~~~~(*)$ $(b)\implies f(x)=x$ has at most one solution in the intervals $(-1,0)$ and $(0,+\infty).$ Let $f(z)=z$ for some $z\in (-1,0).$ Then $(*)\implies f(z^2+2z)=z^2+2z \implies z^2+2z=z$ since $z^2+2z=(z+1)^2-1\in (-1,0).$ Hence, $a=-1$ or $a=0,$ impossible. One can get a similar contradiction for $f(x)=x$ for $x \in (0,+\infty).$ It follows from $(*)$ that $x+(1+x)f(x)=0 \implies f(x)=\frac{-x}{1+x} ~~\forall x\in S,$ which fits. $\square$

$P(0,0)\implies f(f(0))=f(0)$ so $f$ has a fixed point. Let $w$ be any fixed point of $f$. $P(w,w)\implies f(w^2+2w)=w^2+2w$. Thus if $w$ is a fixed point then $w^2+2w$ is also a fixed point. Consider if $w\neq 0$, we divide the value of $w$ into two cases. If $-1<w<0$ then $-1<w^2+2w<w<0$ which is absurd. If $w>0$ then $w^2+2w>w>0$ which is absurd. Hence, $w$ must be $0$. $P(x,x)\implies f(x+(x+1)f(x))=x+(x+1)f(x)$. $$x+(x+1)f(x)=0\implies \boxed{f(x)=-\frac{x}{x+1} \ \forall x\in\mathbb{R}_{>-1}}$$which clearly works.

Let $P(x,y)$ denote the given assertion. $P(0,0): f(f(0))=f(0)$. $P(x,0): f(x+f(0)+xf(0))=f(x)$. $P(x,f(0)): f(x+f(0)+xf(0))=f(0)+f(x)+f(0)f(x)$. So $f(0)+f(0)f(x)=f(0)(f(x)+1)=0$ for any $x$. Since $f(x)>-1$, we have $f(0)=0$. $P(0,x): f(f(x))=x$. Suppose $k$ is a fixed point of $f$. $P(k,k): f(k^2+2k)=k^2+2k$. Note that $\frac{f(k)}{k}=1$ and $\frac{f(k^2+2k)}{k^2+2k}=1$. So by condition b, $k$ and $k^2+2k$ can't be both positive or both negative. However, if $k$ is negative, then since $k>-2$, $k^2+2k = k(k+2)$ is negative also. If $k$ is positive then $k^2+2k$ is also positive. If $k=0$ then $k^2+2k=0$. Thus, either $k$ and $k^2+2k$ are both positive or both negative, or $k=0$, which implies that $0$ is the only fixed point of $f$. $P(x,x): f(x+f(x)+xf(x))=x+f(x)+xf(x)$. So for each $x$, $x+f(x)+xf(x)$ is a fixed point, so \[x+f(x)+xf(x)=0\implies f(x)(x+1)=-x\implies \boxed{f(x)=-\frac{x}{x+1}},\]which works.

Gerhard Woeginger wrote: Suppose $f(a) = a.$ Then putting $x = y = a$ in the relation given, we get $f(b) = b$ where $b = 2a + a^2.$ If $ -1 < a < 0,$ then $ -1 < b < a.$ But $f(a)/a = f(b)/b.$ Contradiction. Similarly, if $a > 0,$ then $ b > a,$ but $f(a)/a = f(b)/b.$ Contradiction. So we must have $a = 0.$ But putting $x = y$ in the relation given we get $ f(k) = k$ for $k = x + f(x) + xf(x).$ Hence for any $x$ we have $x + f(x) + xf(x) = 0$ and hence $f(x) = -\tfrac{x}{x+1}.$ Finally, it is straightforward to check that $f(x) = -\tfrac{x}{x+1}$ satisfies the two conditions. Nice solution. Mr. Woeginger orz EDIT: Did latex.