I've got a question what are the neccesary conditions to apply this method since if I use this to solve $f(f(x))=x$ which is equivalent to $a_n+2=a_n$ with $a_0=x$ I get a unique solution $f(x)=x$, but the function $f(x)=1/x$ is also a valid solution.

Ramanujan1057894736842 wrote: I've got a question what are the necessary conditions to apply this method since if I use this to solve $f(f(x))=x$ which is equivalent to $a_n+2=a_n$ with $a_0=x$ I get a unique solution $f(x)=x$, but the function $f(x)=1/x$ is also a valid solution. You won't get $f(x)=x$. You'll get $f^{2n}(x)=x$ with no information about $f(x)$ (here $f^k(x)$ denotes $k$ times application of $f$)

Fix $x$, and let $a_n = f^n(x)$ (the $n$-th iterate of $f$). Then, \[a_{n+2} +a.a_{n+1} - b(a+b)a_n = 0,\]which has characteristic polynomial \[x^2+ax-b(a+b),\]which has roots $-a-b$ and $b$. Hence, \[a_n = u.(-a-b)^n+v.b^n,\]with $u,v$ constant. Therefore, \[a_n = \frac{xb-f(x)}{a+2b}(-a-b)^n+\frac{(a+b)x+f(x)}{2b+a}b^n,\]which, unless $f(x) = bx$, is negative for some large $n$. Therefore, $f(x) = bx, \ \forall x$, which is a solution.

Define $f(x_{k-1})=x_k.$ Then the original equation becomes \begin{align*}x_{k+2}=b(a+b)x_k-ax_{k+1} \\ \implies x_k=c_1b^k+c_2(-a-b)^k,\end{align*}where $c_1$ and $c_2$ are non-negative constants. We also have $x_0=c_1+c_2,$ and $x_1=c_1b-c_2(a+b).$ Notice that $c_2=0$ for $x_k$ to be non-negative. Thus $$x_0=c_1\implies f(x_0)=x_1=bc_1=bx_0.$$Hence the solution is $f(x)=bx ~~\forall x,$ which fits.

clearly, $f(x)=bx$ is a solution, as $b^2x+abx=b(a+b)x$ it suffices to prove that it is the only one note that $f(f(x))=(ab+b^2)x-af(x)$ for some $c_0\ge 0$, let $c_n=f^n(c_0)$ so $c_1=f(c_0)$, $c_2=f(f(c_0))$, $c_3=f(f(f(c_0)))$, $\dots$ then $c_{n+2}+ac_{n+1}-(ab+b^2)c_{n}=0$ since $x^2+ax-(ab+b^2)$ factors as $(x-b)(x+a+b)$, $c_n=Ab^n+B(-a-b)^n$ for some $A$ and $B$. we see that $A+B=c_0$ and $Ab-B(a+b)=c_1$ $Ab+Bb=bc_0$ so $bc_0-c_1=B(a+2b)$ so $B=\frac{bc_0-c_1}{a+2b}$ and $A=\frac{ac_0+bc_0+c_1}{a+2b}$ FTSOC, assume that $c_1\ne bc_0$. clearly we must have that $A>0$ because $c_0, c_1\ge 0$ and $c_0=c_1=0$ is not allowed. it suffices to prove that $Ab^n+B(-a-b)^n<0$ for some sufficiently large $n$ since $A, b>0$, we can divide both sides by $Ab^n$ to get $$\frac{B}{A}(-\frac{a}{b}-1)^n<-1$$ since $a,b>0$, the magnitude of $-\frac{a}{b}-1$ is more than $1$ let $X=\frac{a}{b}+1$. then let $Y=\lceil\max(69420, \log_X(|\frac{A}{B}|+69420))+10^{69420}\rceil$ then one of $n=Y+69420$ or $n=Y+69421$ will satisfy the inequality, which is a contradiction as $f$ only outputs nonnegative reals, as desired. therefore, $f(x)=bx$ for all $x$ is the only solution.