Do not mention that a very similar problem: $\frac{a^2+b^2}{ab-1}$ appeared not long ago in a USA TST. But it's very interesting what happens with this problem after you start solving it. It has a step curiously difficult. A very nice problem, like all the other problems given in this very difficult TST.

I think I have an idea about what a solution might look like. We consider ordered pairs $(a,b)$ s.t. $a\leq b$. We can go from the pair $(a,b)$ to a "smaller pair" $(a',b')=(\frac{a+b+a^3}{ab-1},a)$ until we eventually reach a "dead-end", i.e. a pair which can no longer become smaller by the above transformation. A simple calculation shows that $f(a,b)$ is invariant wrt the transformation mentioned, so the value we're interested in remains the same. I also think that the "dead-end", as I've called it, must be one of the pairs $(1,2)$, $(1,4)$ and $(2,2)$. By analyzing the 3 cases we obtain $f(a,b)\in \{7,4\}$ (it's 7 in the first two cases and 4 in the third case). I don't even know if this is th correct answer, let alone the correct idea for a solution, but it looks like this at 2 AM .

The problem was proposed at the upmentioned romanian test, but, no matter the fact that the idea of the problem is well-known, at least for people which studied a little bit the questions from past IMOs (especially IMO '88 ), only two contestants have managed to master the problem you are right, Grobber ... the good idea, and also the good result. some more computations for clarification are required, but that's the main idea.

Well, I suppose I don't have to do the actual computations (I pity the contestants for that, as I often lose my interest in a problem after getting the main idea ).. Do you know who proposed the problem?

n As far as I know, it was proposed by Mircea Becheanu.

case1:$a=b$ $f(a,b)=\frac{3a^2}{a^2-1}=3+\frac{3}{a^2-1}\in\mathbb{N}$,we have $a^2-1=3$.Thus $a=2$.i.e. $(a.b)=(2,2)$,yielding $f(a,b)=4$. case2:$a\neq b$ By symmetry of $a$ and $b$,WLOG $a>b\geqq1$,so we can set $a\geqq b+1$,thus we have $f(a,b)=\frac{a^2+ab+b^2}{ab-1}=1+\frac{a^2+b^2+1}{ab-1}\cdots\ $ (☆) $\leqq1+\frac{a^2+(a-1)^2+1}{ab-1}=1+\frac{2\left[(a-1)a+1\right]}{ab-1}\in\mathbb{N}$,since $(a-1)a+1$ is odd, we obtain $2|ab-1$, i.e. $ab-1=1,2$,yielding $(a,b)=(1,2),(1,3)$ For $(a,b)=(1,2),f(a,b)=7$ and $(a,b)=(1,3)=\frac{3}{2}$, in addition to from （☆） in using A.M$\geqq$ G.M and $a\geqq2,b\geqq1$,we obtain $f(a,b)\geqq 1+\frac{2ab+1}{ab-1}\geqq 1+2+\frac{3}{ab-1}>3$. Consequentry,The value of $f(a,b)$ is $4,7$. kunny

you are so far from reality kunny....

kunny´s solution is wrong.. grobber, i don´t undestand your solution. can you help me?

Neither do I...I can't get how the procedure used by Grobber leads to the "dead-end"...and why this very procedure is justified in this case. But still another question: Could we in some way apply...or is there at least a connection to...Diophantine equations of the second order here: ax^2+bxy+cy^2=k? I looked it up at MathWorld and became curious about a possible connection. Cheers!

Valentin Vornicu wrote: Let $a,b$ be two positive integers, such that $ab\neq 1$. Find all the integer values that $f(a,b)$ can take, where \[ f(a,b) = \frac{ a^2+ab+b^2} { ab- 1} . \] i've seen this problem approximately a year ago try to take a minimal and then show that there is x which is less than a but it's pretty hard i think another way is to try to make it with the help of quadratic equation

Valentin Vornicu wrote: Let $a,b$ be two positive integers, such that $ab\neq 1$. Find all the integer values that $f(a,b)$ can take, where \[ f(a,b) = \frac{ a^2+ab+b^2} { ab- 1} . \] Let for some integer $n$ we have $n=\frac{a^2+b^2+ab}{ab-1}$. Suppose $a>b>2$. Then we have $a^2+b^2+ab=nab-n$, $a^2+a\cdot b(1-n)+b^2+n=0$. This equation has the second root (with respect to $a$), call it $a'$. It's easy to prove that $0<a'<a$. We can repeat this while both $a,b>2$. Hence all possible values of $n$ can be obtained while one of the numbers (i.e. b) is 0, 1 or 2, which is easy to handle.

Some Other questions with the same idea : 1: Find all natural numbers x,y,z such that x^2 + y^2 + z^2 = 3xyz 2: Suppose that m is a given natural number ,Show that there exist infinitly pairs (x,y) of natural numbers such that xy | x^2 + y^2 + m 3: For which natural numbers n we can find integers a,b,c,d such that: n = (a^2 + b^2 + c^2 + d^2)/(1 + abcd) Good luck.

armanf wrote: Some Other questions with the same idea : 1: Find all natural numbers x,y,z such that x^2 + y^2 + z^2 = 3xyz 2: Suppose that m is a given natural number ,Show that there exist infinitly pairs (x,y) of natural numbers such that xy | x^2 + y^2 + m 3: For which natural numbers n we can find integers a,b,c,d such that: n = (a^2 + b^2 + c^2 + d^2)/(1 + abcd) Good luck. for the 2nd, saying WLOG that x is the greatest, take the couple $(x',y')=(\frac{y^2+m}x,x'$

I still can't find the complete solution I'm having troubles understanding why the new pair is "smaller" than the previous one. In IMO '88, that's clear, but here Does anyone dare to help me?

can anyone kindly give a complete solution?i am trying something like IMO 88 but it seems its not working here...

Of course Vieta-Jumping is working here See my proof in the attachment.

thanks felixD

It's easy, not? $a^2+b^2+ab$ is divisible by $ab-1$, so $a^2+b^2+2ab-1$ is also divisible by $ab-1$. $(a+b-1)(a+b+1)$ is divisible by $ab-1$. The rest is not so difficult.

With Alan Chen, Alexandru Girban, Anushka Aggarwal, Derek Liu, Dhrubajyoti Ghosh, Eric Shen (USA), Gene Yang, Jeffrey Kwan Paul Hamrick, Sean Li, Zifan Wang: Suppose that \[ k = \frac{a^2+ab+b^2}{ab-1} \iff a^2 - b(k-1) \cdot a + (b^2+k) = 0. \]We contend either $k=4$ or $k=7$. These are achieved by $f(2,2) = = 4$ and $f(2,1) = 7$. Now fix $k$ and suppose $(a,b)$ is a minimal pair for that $k$. If $a=b$ we get $f(a,a) = \frac{3a^2}{a^2-1} = 3 + \frac{3}{a^2-1}$ which easily implies $a=2$ and hence $f(a,a) = 7$. Else, assume $a > b$ and note that if $(a,b)$ is a solution then by Vieta jumping we get a neighboring pair \[ (a,b) \rightarrow \left( \frac{b^2+k}{a}, b \right) = \left( (k-1) \cdot b - a, b \right). \]This forces $\frac{b^2+k}{a} \ge a$, so $k \ge a^2-b^2$. A calculation then gives \begin{align*} \frac{a^2+ab+b^2}{ab-1} &\ge a^2-b^2 \\ \iff a(2a+b) &\ge ab(a-b)(a+b) \\ \iff 2a+b &\ge b(a-b)(a+b) \\ \implies 2 &> b(a-b). \end{align*}This can only occur for $(a,b) = (2,1)$ hence $f(2,1) = 7$. Remark: The Vieta chains for $k=4$ and $k=7$ are given respectively by \[ (2,2) \xleftrightarrow{a^2-6a+8} (2,4) \xleftrightarrow{a^2-12a+20} (10,4) \longleftrightarrow \dots \]and \[ \dots \longleftrightarrow (23,4) \xleftrightarrow{a^2-24a+23} (4,1) \xleftrightarrow{a^2-6a+8} (2,1) \xleftrightarrow{a^2-12a+11} (11,2) \xleftrightarrow{a^2-66a+128} (64,11) \longleftrightarrow \dots. \]The proof above can be phrased conceptually as saying that $(2,1)$ and $(2,2)$ are the unique pairs (across all $k$) which do not have a neighbor of lesser sum.

I've no clue what exactly I'm doing here, new to Vietta jumping, but, well... For $a \neq b$, say $a>b$. Letting $f(a,b)=k \implies a^2+b^2+ab=kab-k \implies a^2+ab(1-k)+(b^2+k)=0$. Say $(a,b)$ is the pair for the particular $k$ that is an output of $f$ such that $b$ is minimal. Now say $a_0$ is another root of $x^2+xb(1-k)+(b^2+k)=0$, and clearly $a_0 \geq b$. Assume $a_0<a \implies a>a_0 \geq b$. Notice $aa_0=b^2+k \implies b^2+k>ab \implies k \geq b(a-b)=ab-b^2$. Also $a+a_0 =b(1-k) \implies 2a>b(k-1) \implies \frac{2a}b +1>k \geq ab-b^2 \implies 2a+b>ab^2-b^3 \implies \frac{b^3+b}{b^2-2} \geq a$. As $a \geq b+1$, this implies $b^3+b \geq (b^2-2)(b+1)=b^3+b^2-2b-2 \implies 0 \geq b^2-3b-2 \implies b=1,2$ or $3$. Thus we have 4 cases to deal with: $b=1 \implies a-1 \mid a^2+a+1 \implies a-1 \mid a^2+a+1-a^2+1 \implies a-1 \mid 3 \implies a=4$ or $a=2$. $b=2 \implies 2a-1 \mid a^2+2a+4=2a^2+4a+8-a(2a-1)=5a+8=10a+16-5(2a-1)=21 \implies a=1,2,4,11$. $b=3 \implies 3a-1 \mid 91=13 \cdot 7 \implies$ Contradiction. $a=b \implies a^2-1 \mid 3a^2 \implies a^2-1 \mid 3 \implies a=2=b$. Checking all these cases give the only values of $k=f(a,b)$ as $4$ or $7$.

maybe i make mistake but if my solution be true it mean problem is too easy for TST. $ f(a,b) = \frac { a^2+ab+b^2} { ab- 1} . $then should $\frac { (a+b)^2-1} { ab- 1} $ be a integer. so we can suppose that $a+b=t$ so: $t^2-1=m(ta-a^2-1)$ then: $t^2-(ma)t+(m-1+ma^2)=0$ then we can check the possible roots and get solution.

We operate on the function $g(a,b) = f(a,b)-1=\frac{a^2+b^2+1}{ab-1}$. We first resolve some edge-cases. Claim: If $b=1$, then $g(a,b)=6$. Solution: Note that \[g(a,1)=\frac{a^2+2}{a-1}=a+1+\frac{3}{a-1},\]so we get $a\in \{2,4\}$ and $g(a,1)=6$. $\fbox{}$ Claim: If $a=b$, then $g(a,b)=3$. Solution: Note that \[g(a,a)=\frac{2a^2+1}{a^2-1} = 2+\frac{3}{a^2-1}.\]This implies $a=2$, so $g(a,b)=3$. $\fbox{}$ We can now freely assume $a>b>1$ WLOG. Set $a/b=r$ so we get \[g(a,b)=\frac{a^2+b^2+1}{ab-1}=\frac{r^2b^2+b^2+1}{rb^2-1} = r+\frac{b^2+r+1}{rb^2-1}=r+\frac{1}{r}+\frac{r+1+\frac{1}{r}}{rb^2-1} \le r+\frac{1}{r}+\frac{r+1+\frac{1}{r}}{4r-1}.\]Then, note the inequality \[4r-1>3r >r+1+1>r+1+\frac{1}{r},\]so we must have \[g(a,b)=\left\lceil r+\frac{1}{r}\right\rceil < r+\frac{1}{r}+1<r+2\]because \[\frac{r+1+\frac{1}{r}}{4r-1}<1\]and $g(a,b)\in\mathbb{Z}$. Now, we use bruteforce to make Vieta Jumping work; we take some choice of $(a,b)$ and either reduce $|a|+|b|$ or get stuck at some particular points. WLOG assume $a>b$ as we already dealt with the other cases. Let $g(a,b)=k$. We can assume $k\not\in\{3,6\}$ as we have already found constructions for those $k$. We write \[a^2+b^2+1=kab-k.\]Rewrite this as \[a^2-(kb)a+b^2+k+1=0.\]Thus, the other root of this quadratic in $a$ is \[\frac{b^2+k+1}{a} < \frac{b^2+k+1}{b(k-2)} = \frac{b}{k-2}+\frac{k+1}{b(k-2)}.\]We take the time to note that as $k\ne 3$ and $k\ge r+\frac{1}{r}>2$, we have \[\frac{b^2+k+1}{a}<\frac{b}{2}+\frac{k+1}{2(k-2)} = \frac{b}{2}+\frac{1}{2}+\frac{3}{2(k-2)}\le \frac{b}{2}+\frac{1}{2}+\frac{3}{4} = \frac{b}{2}+\frac{b}{2}+\frac{1}{4}=b+\frac{1}{4}.\]As the latter expression is an integer and cannot be $b$ (as that would give $k=3$, which is disallowed), we get that $|a|+|b|$ is monotonically decreasing when we Vieta Jump, absurd. Thus, $k\in \{3,6\}$ are the only possibilities and so $f(a,b)\in \{4,7\}$.

Let's look at $g(a, b)=f(a, b)-1=\frac{a^2+b^2+1}{ab-1}.$ Note that by $\text{AM-GM}$ we have $g(a, b)>2.$ Consider the solution $(a_0, b_0)$ with minimal sum, for which $g(a_0, b_0)=k$ is an integer. Assume WLOG $a_0\geq b_0.$ Consider the function $h(a)=a^2-kb_0a+b_0^2+k+1.$ We know that there is a number $a_1\in \mathbb{R},$ such that $$h(a_1)=0, a_0+a_1=kb_0, a_0a_1=b_0^2+k+1.$$From here we easily deduce $a_1\in \mathbb{Z}_{> 0}$ and therefore it must be the case $a_1\geq a_0.$ Case 1: Assume that $b_0\geq 2$ and recall that $k\geq 3.$ We obtain $$h(b_0)=b_0^2(2-k)+k+1\leq 4(2-k)+k+1=9-3k\leq 0,$$so we must have $a_1\geq b_0\geq a_0\geq b_0,$ so $a_0=b_0$ and now by simple polynomial division we obtain $(a_0, b_0)=(2, 2),$ and so $f(a, b)=g(a, b)+1=4.$ Case 2: Let $b_0=1.$ We then have $a_0-1\mid a_0^2+2,$ and so $a_0\in \{2, 4\}.$ In both cases we deduce $f(a, b)=g(a, b)+1=7.$ $\blacksquare$

We claim the only possible integer values are $4$ and $7$. First, note that when $a=b$, the fraction becomes $\frac{3a^2}{a^2-1} = 3 + \frac{3}{a^2-1}$ which is only an integer for $a=2$. This yields $f(a,b)=4$. Now, assume that $a>b>0$. We will use Vieta jumping. We have \[ \frac{a^2+ab+b^2}{ab-1} = k \implies a^2 - (k-1)b \cdot a + b^2 + k .\]Therefore, we can jump from the pair $(a,b)$ to $\left(\frac{b^2+k}{a},b)\right)$. This process always produces a smaller pair when $\frac{b^2+k}{a} < a$ or when $k < a^2 - b^2$. In this case we have, \[ \frac{a^2+ab+b^2}{ab-1} < a^2 - b^2 \iff \frac{2a+b}{a+b} < b(a-b) \]Note that $b(a-b) \ge 2 \ge \frac{2a+b}{a+b}$ when $b \ge 2$. Therefore, the only case when this breaks is when $a=1$ and $b = 2$. This yields $f(a,b) = 7$, so we are done.

The only possible values of $f(a, b)$ are $4,7$, which can be obtained with $(a,b) = (2,1), (2,2)$ respectively. Let $k = \frac{a^{2} + ab + b^{2}}{ab-1}$. Then, we have $a^{2} - a(kb-b) + (k+b^{2}) = 0$. Suppose $a>b>0$ is a solution. Then,$(\frac{k+b^{2}}{a}, b)$ is also a solution. Observe that for $b\geq 2$, or $a-b\geq 2$ then we have \[2a+b < b(a-b)(a+b) = a^{2}b - b^{3} \Rightarrow 2a^{2} + ab < a^{3}b - ab^{3}\]\[\Rightarrow a^{2} + ab + b^{2} < a^{3}b - ab^{3} - a^{2} + b^{2} \Rightarrow a^{2} > \frac{a^{2} + ab + b^{2}}{ab-1} + b^{2} = k+b^{2}\]Therefore, for all $b\geq 2$ or $a-b\geq 2$, when we flip the larger of $(a,b)$, it always decreases it. Thus, we need to check when $a = b$, or when $b = 1$ and $a-b < 2$. When $a = b$, we need $\frac{3a^{2}}{a^{2} -1}$ is an integer, or $3 + \frac{3}{a^{2} - 1}$ is an integer. The only case where this is possible is $a = b = 2$. When $b = 1$ and $a-b < 2$, this is when $a = 2$. For both these cases, flipping the larger one does not decrease, so all solutions $(a,b)$ can be continuously flipped to reach one of these two. Since $k$ is constant when flipped, the possible values of $k$ are $f(1,2), f(2,2) = 4, 7$.

Rewrite the problem as $\tfrac{a^2+b^2+1}{ab-1}=k \in \mathbb{Z}$ which is the same as $a^2-(bk)a+b^2+k+1=0$; given any solution $(a,b)$ with $a \ge b$ we Vieta jump to $(\tfrac{b^2+k+1}{a},b)$. It suffices to find all minimal solutions, which have $\tfrac{b^2+k+1}{a} \ge a$, so $k \ge a^2-b^2-1$. Letting $a=b+t$ for some $t \ge 0$, we have $$\frac{a^2+b^2+1}{ab-1}=\frac{2b^2+2bt+t^2+1}{b^2+bt-1} \ge a^2-b^2-1=2bt+t^2-1$$which is equivalent to $$(2t)b^3+(3t^2-3)b^2+(t^3-4t)b-2t^2 \le 0.$$When $t=0$ we see that all $b$ work, while when $t=1$ we can easily see that we must have $b=1$. There's no solutions when $t \ge 2$ for size reasons (for instance, $(2t)b^3+(3t^2-3)b^2+(t^3-4t)b-2t^2 \ge 3t^2-3-2t^2 >0$). When $t=0$ we have $a=b$, and so $\tfrac{a^2+b^2+1}{ab-1} =\tfrac{2a^2+1}{a^2-1} \in \mathbb{Z} \implies \tfrac{3}{a^2-1} \in \mathbb{Z} \implies a=b=2$. When $t=1,b=1$ we get $a=2$ which indeed works. So the only minimal pairs with $a \ge b$ are $(a,b)=(2,2),(2,1)$, which in the context of the original problem give $f(a,b)=4,7$ respectively.

WLOG $a\ge b$. Suppose $k=f(a,b)-1=\tfrac{a^2+b^2+1}{ab-1}$. Then \[ a^2-(kb)a+(b^2+1+k) = 0. \]So if $(a,b)$ works, then $(kb-a,b)$ works. Also, if $(a,b)$ works, then $(a,ka-b)$ works by symmetry. Consider the pair $(a,b)$ with minimal $a+b$ that works. Then we must have $kb-a \ge a$, so \begin{align*} kb-a \ge a &\implies k=\frac{a^2+b^2+1}{ab-1} \ge \frac{2a}{b} \\ &\implies a^2b+b^3+b \ge 2a^2b - 2a \\ &\implies 2a+b\ge a^2b-b^3=b(a-b)(a+b) \\ &\implies 2 > b(a-b). \end{align*}Therefore, $(a,b)=(2,1)$ or $a=b$. In the latter case, $f(a,b)=7$, and in the latter case, $f(a,a)=3 + \frac{3}{a^2-1}$, which is an integer if and only if $a=2$, in which case $f(2,2)=7$. The answer is 3 and 7.

First, setting $b=1$ gives $a^2+a+1-(a+2)(a-1)=3$ must be a multiple of $a-1$, forcing $a=2,4$, both of which cases gives $f(a,1)=7$. Setting $a=b$ similarly gives that $3a^2-3(a^2-1)=3$ is a multiple of $a^2-1$, similarly forcing $a=2$, and $f(2,2)=4$. Now, we claim that these are the only possible values. WLOG assume $a>b>1$ from here onwards, and let $(x,y)$ be a solution to $f(x,y)=k$ such that $x+y$ is minimized. We have $x^2+xy+y^2=k(xy-1)\rightarrow x^2-(k-1)xy+y^2+k=0$; treat this as a quadratic in $x$, and with Vieta's formulas, we know that the other solution is $ky-y-x$, which must be equal to $\frac{y^2+k}{x}$. The former gives that it is an integer, and the latter gives that it is positive, and therefore another valid solution; for a contradiction to not arise, we need $\frac{y^2+k}{x}\ge x$, or $k\ge x^2-y^2$. $x^2+xy+y^2\ge (x^2-y^2)(xy-1)$ can be rearranged to $2x+y+y^3\ge x^2y$. Now, let $\frac{x}{y}=r$; the inequality can be again rewritten as $2r+1\ge (r^2-1)y^2$, from which we can solve $r\le \sqrt{\frac{y^4+y^2+1}{y^4}}+\frac{1}{y^2}<\sqrt{\frac{y^4+2y^2+1}{y^4}}+\frac{1}{y^2}=\frac{y^2+2}{y^2}\le \frac{y+1}{y}$, which makes it impossible for $x$ to be an integer.

We claim that $\boxed{f(a,b) = \frac { a^2+ab+b^2} { ab- 1} \in [4,7]}$ WLOG,assume that $a \ge b$,and let $k=f(a,b)-1=\tfrac{a^2+b^2+1}{ab-1}$. Expanding it,\[ a^2-(kb)a+(b^2+1+k) = 0. \]So if $(a,b)$ works, then $(kb-a,b)$ works. Also, if $(a,b)$ works, then $(a,ka-b)$ works by symmetry. By Vieta Jumping,consider the pair $(a,b)$ with minimal sum:- \begin{align*} kb-a \ge a &\implies k=\frac{a^2+b^2+1}{ab-1} \ge \frac{2a}{b} \\ &\implies a^2b+b^3+b \ge 2a^2b - 2a \\ &\implies 2a+b\ge a^2b-b^3=b(a-b)(a+b) \\ &\implies 2 > b(a-b). \end{align*}Therefore, $(a,b)=(2,1)$ or $a=b$. In the former case, $f(a,b)=7$, and in the latter case, $f(a,a)=3 + \frac{3}{a^2-1}$, which is an integer if and only if $a=2$, in which case $f(2,2)=4$.$\blacksquare$

The answer is $f(a,b) \in \{4,7\}$. We can check that $f(2,1)=7$ and $f(2,2)=4$. Further, if $a=b$, then $$f(a,b)=\frac{3a^2}{a^2-1}=3+\frac{3}{a^2-1},$$so $a=2$ and $f(a,b)=4$. We can also check that if $\min\{a,b\}\leq 2$ we only have $f(a,b) \in \{4,7\}$. Now suppose that $k \neq 4,7$ is in the range of $f$, and pick $(a,b)$ such that $f(a,b)=k$, $a>b$, and $a+b$ is minimal, so $b>2$. Treating $b$ as a constant, $f(a,b)=k$ rewrites as the following quadratic in $a$: $$a^2-(k-1)ba+(b^2+k)=0.$$By Vieta jumping, $(\tfrac{b^2+k}{a},b)$ is also a solution, and by minimality we must have $$\frac{b^2+k}{a}\geq a \implies k \geq a^2-b^2 \geq a^2-(a-1)^2=2a-1.$$This forces \begin{align*} \frac{a^2+ab+b^2}{ab-1} &\geq 2a-1\\ 3a^2 &\geq 2a^2b-ab-2a+1\\ 3a^2 &> 2a^2b-3a^2\\ 3a^2 &> a^2b, \end{align*}which evidently only holds for $b \leq 2$, contradiction. Thus such a $k$ does not exist, so $f(a,b) \in \{4,7\}$ as desired. $\blacksquare$ Sprites wrote: We claim that $\boxed{f(a,b) = \frac { a^2+ab+b^2} { ab- 1} \in [4,7]}$ btw this is saying that $4 \leq f(a,b) \leq 7$

The answer is $f(a, b) \in \{4, 7\}$. Fix $k = \frac{a^2+ab+b^2}{ab - 1}$. Then the equation simplifies to $$a^2+(b-kb)a + b^2+k = 0.$$This means that if $(a, b)$ yields a particular value of $k$, then $\left(\frac{b^2+k}a+b\right)$ yields this value of $k$ too. WLOG $a > b$. Claim. If $a, b \geq 2$, $b^2+k < a^2$ unless $a=b$. Proof. The inequality simplifies to \begin{align*} a^2b-a &> b^3+a+b \\ b(a+b)(a-b)& > 2a+b \\ ab+b^2 &> 2a+b \end{align*}as $a-b > 0$. On the other hand the final display is clearly true. $\blacksquare$ This means that all values in the range of $f(a, b)$ can be generated by $f(a, a)$ or $f(\leq 2, \leq 2)$. The only such values are $4, 7$, as needed.

Answer: $4, 7$. If $\min(a, b) \le 3$, then one can check that all pairs $(a, b)$ such that $f(a, b)$ is an integer are $(4, 1), (2, 1), (11, 2), (4, 2), (2, 2)$ and we can check the case $a = b$. Let $(a, b)$ be a pair such that $f(a, b)$ is an integer and $a > b > 3$ and let $c = f(a, b)$. Then we have $a^2 + ab + b^2 = cab - c$, or equivalently $a^2 + a(b - cb) + (b^2 + c) = 0$. Let $a_1$ be a root of $x^2 + x(b - cb) + (b^2 + c) = 0$ other that $a$. Then $a_1 + a = cb - b$ and $aa_1 = b^2 + c$. Thus $a_1$ is an integer and $a_1 > 0$. We'll prove that $b > a_1$. Assume not. Then we get $b \le a_1 = \frac{b^2 + c}{a}$, or equivalently $c \ge ab - b^2$. Therefore $a^2 + ab + b^2 \ge (ab - 1)(ab - b^2)$. By expanding, we get $a(2b + b^3) \ge a^2(b^2 - 1)$, so $2b + b^3 \ge a(b^2 - 1) \ge (b + 1)(b^2 - 1)$. Thus we get $0 \ge b^2 - 3b - 1$. Hence $b \le 3$, a contradiction. Thus we have $a_1 < b$. Thus if we replace $(a, b) \to (a_1, b)$, then $\min(a, b)$ decreases. Therefore repeating this process until $\min(a, b) \le 3$, we obtain $c = 4$ or $7$. This completes proof. $\blacksquare$

First notice the only solution for $a=b$ is $(2,2)$, so assume WLOG $a>b$. Letting $f(a,b)=k$, our equation can be rewritten as \[a^2-(bk-b)a+(b^2+k) = 0.\] By Vieta's, if $(a,b)$ is a valid solution, then $\left(\frac{b^2+k}{a},b\right)$ is also a valid solution. To show this procedure always leads to smaller pairs, note \begin{align*} \frac{b^2+k}{a} < a &\iff k < a^2-b^2 \\ &\iff \frac{a^2+ab+b^2} < a^2-b^2 \\ &\iff 2a+b < (a-b)(ab+b^2) \end{align*} always holds unless $b=1$, where we find the only valid values of $a$ are 2 and 4, both of which give $k=7$. Hence we stop our algorithmic decreasing only when One of the values is 1, meaning the pair is $(2,1)$ or $(4,1)$, leaving $k = \boxed{7}$ Both values are equation, meaning the pair is $(2,2)$, leaving $k = \boxed{4}$. $\blacksquare$

Let $f(a,b) = \frac{a^2+ab+b^2}{ab-1} = k$, where k is an integer. We get the quadratic equation $a^2+(b-kb)a+b^2+k = 0$. Now by Vieta jumping we can generate new solutions from (a,b) to $(\frac{b^2+k}{a},b)$. Now WLOG $a \geq b$. If a = b, we need to have that $a^2 - 1 \mid 3a^2$ $\Rightarrow$ $a^2 - 1 \mid 3$ $\Rightarrow$ $a^2 = 4$, since a is a positive integer and $ab - 1 \not \equiv 0$ $\Rightarrow$ a = b = 2 and we get the solution (a,b) = (2,2) and k = 4. Now the neighboring pair $(\frac{b^2+k}{a},b)$ forces $\frac{b^2+k}{a} \geq a$ $\Rightarrow$ $k \geq a^2-b^2$. Now plugging that in the original equation for f(a,b) we get $a^2+b^2+ab \geq (ab-1)(a^2-b^2)$, which is equivalent to $2a^2+ab \geq ab(a^2-b^2)$ $\Rightarrow$ we get that $2>b(a-b)$. Since $a>b$ and a and b are positive integers the only (a,b) that satisfies this condition is (a,b) = (2,1), where k=7 $\Rightarrow$ we got that the only possible k are k=4 when (a,b) = (2,2) and k=7 when (a,b) = (2,1) $\Rightarrow$ f(a,b) = 4;7. We are ready.

Might have overdid it whoops