Motivation: $(\sqrt{k}y-\sqrt{k-1}x)(\sqrt{k}-\sqrt{k-1})^2=\sqrt{k}((2k-1)y+2(k-1)x)-\sqrt{k-1}((2k-1)x+2ky)=\sqrt{k}y'-\sqrt{k-1}x'$. We use this to generate solutions to the equation repeatedly. If $(x,y)$ is a solution, then note that $(x',y')=((2k-1)x+2ky, (2k-1)y+2(k-1)x)$ is a solution. Thus, if there is a solution with $0\le x<\sqrt{a}$, then we can get a solution with $x>\sqrt{a}$ by repeatedly applying the first step $(x,y)\rightarrow (x', y')$. Thus $B\subseteq A$. If $(x,y)$ is a solution, then note that $(x', y')=((2k-1)x-2ky, (2k-1)y-2(k-1)x)$ is a solution. Suppose $(x,y)$ is a solution with $x>\sqrt{a}$. Then $x>y$. We can check that $y'>0$; indeed, $(2k-1)^2y^2-4(k-1)^2x^2=4(k-1)+y^2>0$. Note that $x'<0$ iff $(2k-1)^2x^2<4k^2y^2$, or $x^2<4k$. If $x'$ is positive, note that $x'+y'=x-y<x$, so $x'<x$. Then, repeat the procedure until either $x<\sqrt{4k}$ or $x<y$; in the latter case, we have $x<\sqrt{a}$. Otherwise, $x<\sqrt{4k}$. Then $x'$ is negative, so $x'+y'=|y'|-|x'|=x-y>0$. Thus $|y'|>|x'|$, and now we have a solution to the equation with $x<y$, and once again $x<\sqrt{a}$. It follows $A\subseteq B$. Combining both parts, $A=B$.

Very clever!

Pretty sure that $x, y \in \mathbb{Z}$? Motivation: $k = 2$ is a Pell equation. After proving it for $k = 2$, everything else is straightforward. $k = \frac{x^2 - a}{y^2 - a} \implies (k - 1)x^2 - ky^2 = -a \implies x_1^2 - k(k - 1)y^2 = -a(k - 1)$ where $k - 1 | x_1$. Let's use $x$ in place of $x_1$ for simplicity. Let $(r, s)$ be the fundamental (both positive) solution to the pell equation where $D = k(k - 1)$. In other words, $$r^2 - k(k - 1)s^2 = 1$$and let $(x, y)$ be a solution to $$x^2 - k(k - 1)y^2 = -a(k - 1)$$where $k - 1 | x$. We work in the ring $\mathbb{Z}\left[\sqrt{k(k - 1)}\right]$ and $(a, b)(c, d)$ is assumed to be the multiplication and $(a, b) = a + \sqrt{k(k - 1)}b$. It's obvious that $(x, y)(r, s) = (p, q)$ has norm $-a(k - 1)$ because $N(r, s) = 1$. Additionally, we have that $k - 1 | p$, so $(p, q)$ is a desired solution. This works with any sign combination of $r$ and $s$. So if $(x, y)$ is a solution such that $x < \sqrt{a(k - 1)}$, we can keep on multiplying by $(p, q)$ to get an $x$ that is larger than $\sqrt{a(k - 1)}$. Therefore, it is obvious that $B \subseteq A$. If $x > \sqrt{a(k - 1)}$, then we will show that multiplying by $(-r, s)$ or by $(r, -s)$ works, depending on sign. Multiplying, we have $$(r, -s)(x, y) = (xr - syk(k - 1), \dots)$$Ideally, we want $$|xr - syk(k - 1)| < x \implies (r - 1)x < syk(k - 1) < (r + 1)x$$If that does not happen, we must prove that in the case that the left inequality isn't satisfied, the equation $x^2 - k(k - 1)y^2$ produces a positive integer and in the case that the right inequality isn't satisfied, $x < \sqrt{a(k - 1)}$. We want the ratio of $y$ to $x$ to be as small as possible, so that $x$ is as large as possible. Therefore $$syk(k - 1) \ge (r + 1)x \implies \frac{y}{x} \ge \frac{(r + 1)}{sk(k - 1)}$$we can have $y = \frac{(r + 1)x}{sk(k - 1)}$. So $$x^2 - k(k - 1)y^2 = -a(k - 1) \implies x^2\left(1 - \frac{(r + 1)^2}{k(k - 1)s^2}\right) = x^2\left(\frac{s^2k(k - 1) - r^2 - 2r - 1}{s^2k(k - 1)}\right) = x^2\left(\frac{2r + 2}{s^2k(k - 1)}\right) = a(k - 1)$$Substituting $(k - 1)x$ back into $x$, and rearranging, we have $$x^2 = \frac{aks^2}{2r + 2}$$So we must prove that $ks^2 \le 2r + 2$. Note that $k(k - 1)s^2 = r^2 - 1 \implies ks^2 = \frac{r^2 - 1}{k - 1}$, so we must prove that $\frac{r^2 - 1}{k - 1} \le 2r + 2$. Rearranging, we have that $$r^2 - 2r(k - 1) - 2(k - 1) - 1 \le 0 \iff (r + 1)(r + 1 - 2k) \le 0$$Since both $r$ and $k$ are positive, this is equivalent to $r \le 2k - 1$. But $N(2k - 1, 2) = 1$!, so $r \le 2k - 1$. So $ks^2 \le 2r + 2$. Equality never holds because $a$ is not a perfect square. This proves that the right inequality of $$x \implies (r - 1)x < syk(k - 1) < (r + 1)x$$must hold if $x > \sqrt{a(k - 1)}$. Suppose the left inequality did not hold. We will prove that $N(x, y) \ge 0$. Suppose it isn't. In order to make the norm as small as possible, we'd want $\frac{y}{x}$ to be as large as possible. So we'd want $y = \frac{(r - 1)x}{sk(k - 1)}$. Our goal is to prove that $$1 - \frac{(r - 1)^2}{k(k - 1)s^2} \ge 0$$. Note that $$1 - \frac{(r - 1)^2}{k(k - 1)s^2} = \frac{k(k - 1)s^2 - r^2 + 2r - 1}{k(k - 1)s^2} = \frac{2r - 2}{k(k - 1)s^2} \ge 0 \iff r \ge 1$$And since $r \ge 1$ (well, $r$ is always greater than $1$, really, so the inequality is actually strict), the inequality is true. This finishes the proof that $A = B$. $$\boxed{Q.E.D.}$$

I feel like none of the above equations really show how one would get the superquick solution here. The basic motivation is to just try dumb things (assume things to make it simple). First, the equation rearranges to $a = ky^2 - (k-1)x^2$. Now since the equation is kind of reminiscent, we might pursue a vieta approach of generating more solutions from one (or come up with this by experimentation; more experimentation would actually illustrate the exact form of the new solutions but that might be too much tedious work). So we want to generate a new solution $(x_2, y_2)$ given an existing solution $(x_1, y_1)$. If we did have a new solution, they would satisfy: $$ ky_1^2 - (k-1)x_1^2 = a = ky_2^2 - (k-1)x_2^2$$$$ \Rightarrow k(y_1+y_2)(y_1-y_2)= (k-1)(x_1+x_2)(x_1-x_2) $$ Now we will make assumptions to simplify this equation. Since we want the pairs to be on different sides of $\sqrt a$, we take something that forces $x_1 < y_1 < \sqrt a < y_2 < x_2$ (or the reverse) which is: $y_1 - x_1 = x_2 - y_2 \Rightarrow y_1+y_2= x_1+x_2$ Then our equation simplifies into $ k(y_1-y_2)= (k-1)(x_1-x_2) $. This can be solved for $y_2$ and $x_2$ just like any other system of two equations. The final result is: $$y_2=(2k-1)y_1 - (2k-2)x_1$$$$x_2=2ky_1 - (2k-1)x_1$$ Using these you can easily go from a solution in $A$ to one in $B$, or with some iterations from one in $B$ to one in $A$. So $A=B$, (QED)

I have a somewhat different solution. Here I provide it with the motivation. It is a good idea to Vieta jump, because that transforms the problem into a bounding problem. We would like to Vieta jump for the variable $x$, but alas, we don't have a linear term in $x$, and fixing $y^2$ fixes $x^2$ anyway. But, we can use a trick, which gives us something to work with, and effectively solves the problem Note that $x^2 > a \iff x^2 > y^2$. So, we take two cases. Case 1: $x > \sqrt a$ Let $y = x - d$, where $d, y > 0$. Then, the equation becomes $x^2 - 2kdx + d^2k - a = 0$. Suppose $2kd - x \ge x$. This implies $x \le kd$. Then, $x^2 - a = (2x-d)*kd \ge (2x - d)*x$, implying $-a \ge x(x-d)$, a contradiction. Now, suppose $2kd - x < - \sqrt a$, then $0 = x(x-2kd) + d^2k - a > \sqrt a * (2kd + \sqrt a) + d^2k - a$, which is impossible. Thus, we can obtain a strictly smaller integer solution of the equation, which is greater than $- \sqrt a$, so we actually get one in the range $(- \sqrt a, \sqrt a)$, so we're done. Case 2: $0 < x < \sqrt a$ ($x = 0$ obviously doesn't hold) Let $y = x + d$, where $d, y > 0$. Then, the equation becomes $x^2 + 2kdx + d^2k - a = 0$. Note that the other integer solution of the equation is negative. Suppose $-2kd - x > - \sqrt a$, then $0 = x(x+2kd) + d^2k - a < \sqrt a * (\sqrt a -2kd) + d^2k - a$, implying $d^2 > 4a$. $d^2k - a < 0 \implies a > d^2k > 4ak$, a contradiction. So we get an integer solution of the equation which is smaller than $- \sqrt a$, so we're done.

The main idea is to transform the equation $ky^2-(k-1)x^2=a$ to a pell's equation with a linear transformation, which turns out to be a nice one. It suffices to show that given fixed $k$ and $a$, then the equation has a solution with $x>\sqrt a$ if and only if the quation has a solution with $0<x<\sqrt a$. Now rewrite the equation as $$ky^2-(k-1)x^2=a$$Substitute $y=c+(k-1)d$ and $x=c+kd$, notice that each pair $(x,y)$ corresponds to a unique $(c,d)$. Then the equation becomes \begin{align} c^2-k(k-1)d^2=a \end{align}This is a Pell's equation, meanwhile we have $(2k+1)^2-k(k-1)\cdot 2^2=1$. Hence it has infinitely many solutions. Therefore, if it has a solution with $0<x<\sqrt a$, then there exists a solution to it with $c,d$ sufficiently so that $x=c+kd>\sqrt a$ as desired. Now suppose on the contrary that there exists a solution $(x,y)$ with $x>\sqrt a$ and no solution with $0<x<\sqrt a$. We pick the one with smallest $x$. Notice that $x>y$. Let $N=k(k-1)$. Now $$\frac{c+d\sqrt N}{2k+1+2\sqrt N}=(c+d\sqrt N)(2k+1-2\sqrt N)=c(2k+1)-2dN+(d(2k+1)-2c)\sqrt N$$Therefore $(c(2k+1)-2dN,d(2k+1)-2c)$ is also a solution to $(1)$. Let $c_1=c(2k+1)-2dN$, $d_1=d(2k+1)-2c$, $y_1=c_1+(k-1)d_1$ and $x_1=c_1+kd_1$. CLAIM. $|x_1|<x$ Proof. We have \begin{align*} x_1=c_1+kd_1&=((2k+1)-2k)c+(k(2k+1)-2k(k+1))d\\ &=2kd-c-kd\\ &=2k(x-y)-x\\ &=(2k-1)x-2ky \end{align*}Now since $x>y$ we have $$(2k-1)x-2ky>-x$$Meanwhile, $ky^2-(k-1)x^2>0$, together with $x>y$ we obtain $ky>(k-1)x$, hence $2(k-1)x-2ky>x$. This proves the claim. $\blacksquare$ However, this implies that $(|x_1|,y_1)$ is a solution such that $|x_1|<x$, which contradicts the minimality of $x$, this completes the proof.

Observe that $k= \frac{x^2-a}{x^2-y^2} \iff k(x^2-y^2)=x^2-a \iff (k-1)x^2-ky^2=-a \iff ((k-1)x)^2-k(k-1)y^2=-a(k-1) \quad (\star)$, which is a Pell Equation. Hence, if $k\in B$, since $(\star)$ is a Pell Equation and it has at least one solution, then it has infinitely solutions, so $k \in A$, and therefore $B \subseteq A$. Now, we need to show that if $k \in A$, then $k \in B$. Take $k \in A$. From $(\star)$, there exist $x_1,y_1 \in \mathbb{Z}_{\geq 0}$ with $x_1 \geq \sqrt{a}$ such that $u_1=x_1(k-1)$ and $u_1^2-k(k-1)y_1^2= -a(k-1)$. $\quad (\heartsuit)$ Now, notice that the Pell Equation $m^2-k(k-1)n^2=1$ has minimal solution $(m_0,n_0)=(2k-1,2)$ and $(\heartsuit)$ has a solution $(u_1,y_1) \implies$ its minimal solution $(u_0,y_0)$ satisfies $u_0+\sqrt{k(k-1)}y_0 < 2k-1+2\sqrt{k(k-1)}= (\sqrt{k}+ \sqrt{k-1})^2.$ $\quad (\spadesuit)$ Thus, suppose that $x_0=\frac{u_0}{k-1} > \sqrt{a} \implies u_0^2>a(k-1)^2 \implies$ since $k(k-1)y_0^2=u_0^2+a(k-1) \implies k(k-1)y_0^2 > a(k-1)^2+a(k-1)=ak(k-1) \implies y_0 > \sqrt{a}$ From $(\spadesuit)$, $(\sqrt{k}+\sqrt{k-1})^2 > \sqrt{a}(k-1)+ \sqrt{k(k-1)}.\sqrt{a}=\sqrt{a(k-1)}(\sqrt{k}+\sqrt{k-1}) \implies \sqrt{k}+\sqrt{k-1}> \sqrt{a(k-1)} \implies 2k-1+2\sqrt{k(k-1)} > a(k-1)$. By AM-GM, $4k-2 \geq 2k-1+2\sqrt{k(k-1)}$, so $4k-2 > a(k-1) \implies a < 4+\frac{2}{k-1}$, but since $a$ is not a perfect square $a=2$ or $a=3$, which is easy to lead it to a contradiction (we would have $k<0$). Therefore, $A \subseteq B$, so $A=B$ and we are done. $\blacksquare$

Solved with Alan Bu, Christopher Qiu, Elliott Liu, Jeffrey Chen, Kevin Wu, and Ryan Yang. We Vieta jump. Implicitly assume \(x,y>0\). Let \(c=x+y\) and \(d=x-y\) have the same parity and \(c>d\), so it suffices to show that a solution with \(d>0\) exists if and only if a solution with \(d<0\) exists. Then the equation rewrites as \begin{align*} 4a&=ky^2-(k-1)x^2\\ &=k\left(\frac{c-d}2\right)^2-(k-1)\left(\frac{c+d}2\right)^2\\ &=c^2+d^2-(4k-2)cd\\ \iff 0&=c^2-(4k-2)d\cdot c+(d^2-4a). \end{align*}It follows that a solution \((c,d)\) generates \[(c,d)\mapsto\big(d,(4k-2)d-c\big)=\left(d,\frac{d^2-4a}c\right).\] If we take a minimal positive solution \(c>d>0\), we have \[\frac{d^2-4a}c<\frac{d^2}c<c.\] If we take a maximal negative solution \(c>0>d\); if \(|c|>|d|\), then \[\frac{d^2-4a}c<\frac{d^2}c<c,\]and if \(|c|<|d|\), then \[\frac{c^2-4a}d>\frac{c^2}d>d.\]

Solved with Pujnk Let $y = x-c$ so that the given equation rewrites as $$x^2 - (2ck)x + (c^2k-a) = 0$$ Suppose $x$ is the smallest positive value that satisfies the equation, we will show that if $x > \sqrt{a}$, then we can generate a new smaller pair. WLOG assume $y > 0$ and since $x^2 > a$, we have $c > 0$ See that there is another root to the equation, which is $2ck - x$. Now, we claim $|x-2ck|$ gives a smaller value of $x$. Observe that this fails only when $2ck \ge 2x \implies ck > x$. Suppose this was true. Then, we would have $x^2 - a = kc(2x-c) > x(2x-c) \implies x^2 + a < cx \implies c > x$. But then, this means $x-y > x \implies y < 0$, which is impossible. Similarly, we can deal with the other case, to show that we can generate larger solutions. So, for every $k$ that has a solution for $x^2 > a$ or $x^2 < a$, we have shown there is also a solution for $x$ with the other. So, we indeed have $A = B$, as desired. $\blacksquare$

$k(x^2-y^2) = x^2 - a \Longleftrightarrow 4a = 4ky^2-4(k-1)x^2$, so let $m = x+y, n = x-y$, then $x^2-a$ matches the sign of $n$ and $m$ must be positive. So the equation is $4a = m^2-(4k-2)mn+n^2$, and it is easy to see that m and n have the same parity, so the solution sets form a bijection. Hence, it suffices to show that if this equation has a solution with $mn > 0$ if and only if it has a solution with $mn < 0$. So either $4a = m^2-(4k-2)mn+n^2$ or $4a = m^2+(4k-2)mn+n^2$ has a positive-positive solution then we want to show there is a positive-negative solution. However, by Vieta jumping and infinite descent, we eventually reach a solution that is not positive-positive, finishing. In the special case that we reach $mn = 0$ we must have $n^2 = 4a$, contradicting that $a$ is not a perfect square.

Solved with Sross314. Let $x+y = m, x-y = n.$ The equation rearranges to $m^2-(4k-2)mn+n^2-4a = 0.$ The key idea is $n$ has the same sign as $x-\sqrt{a}.$ Suppose we had a minimal sum pair of positive integers $m,n$ satisfying $m^2-(4k-2)mn+n^2-4a = 0.$ Then we can substitute $m$ with the other integer root of this equation in $m$ equalling $4k-2-m = \frac{n^2-4a}{m} < n,$ and this is negative because $a$ is not a perfect square. So if we have a solution with $x-\sqrt{a} > 0,$ we have one with $x-\sqrt{a} < 0.$ On the flipside, we can do the exact same thing with minimal sum positive integer solutions to $m^2-(2-4k)mn+n^2-4a = 0,$ so if we have a solution with $x-\sqrt{a} < 0,$ we have one with $x-\sqrt{a} > 0$ and this is enough. $\blacksquare$

The equation is equivalent to the Pell Equation\[ky^2 - (k-1)x^2 = a\]Notice that if $(x,y)$ is a solution to this equation, then\[((2k-1)x \pm 2ky, (2k-1)y \pm (2k-2) x)\]is also a solution to this equation, where the $\pm$ signs correspond. If there is a solution with $x<\sqrt{a}$, we can repeatedly apply\[(x,y) \mapsto ((2k-1)x \pm 2ky, (2k-1)y \pm (2k-2) x)\]and get a solution where $x>\sqrt{a}$. Thus, $B\subseteq A$. Claim: If $x>\sqrt{a}$ and $(x,y)$ is a solution to the equation, then $|(2k-1) x - 2ky| < x$ Proof: This is equivalent to\[-x < (2k-1) x - 2ky < x,\]which is equivalent to $0< kx-ky < x$. If $y>x$ was true, then\[ky^2 - (k-1) x^2 > ky^2 - (k-1) y^2 = y^2 > a,\]contradiction. Thus, $kx-ky>0$. It suffices to show $k(x-y) < x$. We have\[ky^2 > (k-1)x^2\implies \sqrt{k}y > \sqrt{k-1} x \]Multiplying both sides by $\sqrt{k}$ gives $ky > (k-1) x $. Rearranging gives $x > kx - ky$, as desired. $\square$ This claim implies that if we have a solution with $x>\sqrt{a}$, then repeating the operation\[(x,y) \mapsto (|(2k-1)x - 2ky|, |(2k-1)y - (2k-2) x| ) \]gives us a solution. So $A\subseteq B$. Thus, $A=B$.

It is only necessary to consider $k$ such that the equation admits a solution. The equation is equivalent to $$ky^2-(k-1)x^2=a.$$Let $x+y=p, x-y=q$, so $p \geq 0$, but we can possibly have $q \leq 0$. Then the equation becomes $$k\left(\frac{p-q}{2}\right)^2-(k-1)\left(\frac{p+q}{2}\right)^2=a \iff p^2-(4k-2)pq+q^2=4a.$$We will focus on the equation on the right for arbitrary $p,q \in \mathbb{Z}$, treating $k$ and $a$ as constants. First note that if $(p,q)$ is a solution then so are $(-p, -q)$ and $(q, p)$. Further if either variable is $0$ then $a$ must equal a square, so assume $p,q$ are nonzero. Further note that if $p$ and $q$ are different parity, then the LHS is odd, but the RHS is always even, so if we have a solution $(p,q)$ to this equation then we can generate a working solution $(x,y)$ to the original equation. By Vieta jumping, if $(p, q)$ is a solution to this equation, then so is $$(p, (4k-2)p-q)=\left(p, \frac{p^2-4a}{q}\right).$$Immediately, if we have some solution $(p,q)$ with $p>0$ and $q<0$, corresponding to a solution $(x,y)$ in the positive integers (this can be easily checked) in the original, then $(4k-2)p-q>(4k-2)p>0$, so we have some solution $(p,q')$ with $p>0$ and $q'>0$ as well. Then the solution $(p,q)$ has $x<y \implies x<\sqrt{a}$ and the solution $(p,q')$ has $x>y \implies x>\sqrt{a}$, so we have $k \in B \implies k \in A$. It remains to deal with the reverse direction. Now we will prove that there do not exist $k$ such that $k \in A$ but $k \not \in B$. To do this, I will prove that there must exist some solution $(p,q)$ to this equation such that $p>0$ and $q<0$ by contradiction. Consider the solution $(m,n)$ to our equation such that $|m+n|$ is minimized. WLOG let $|m|\leq |n|$ (by the fact that $(p,q) \iff (q,p)$) and $m>0$ (by the fact that $(p,q)$ \iff $(-p,-q)$). If there don't exist solutions $(p,q)$ with $p>0$ and $q<0$ (which is true by assumption), then we can Vieta jump from this solution $(m,n)$ to some other solution $(m,n')$ such that $nn'=m^2-4a$. By assumption, $n>0$ as well, so we must have $n' \geq n$ by the minimality of $|m+n|$ (else $n'$ is negative), but then $$m^2-4a=nn'\geq n^2 \implies m^2-n^2\geq 4a>0,$$which is impossible as this implies that $|m|>|n|$. Hence there exists some solution $(p,q)$ with $p>0$ and $q<0$ as desired. Now, given this solution, use $(p,q) \iff (q,p) \iff (-p,-q)$ to WLOG assume that $|p|>|q|$ (if we have $|p|<|q|$ initially, $(-q,-p)$ is a solution as well). This then corresponds to a solution $(x,y)$ to our original equation with $x,y>0$, $x<y$. Since $k$ is positive, $x<y \implies x^2<y^2 \implies x^2<a$, so $x<\sqrt{a}$. Hence if we have $k \in B$ (so there exists some solution $(p,q)$ to our transformed equation), then we can always generate a solution $(x,y)$ to the original equation with $x<\sqrt{a}$, whence $k \in A$. This completes both directions of the problem, so we're done. $\blacksquare$

After with a small hint Let $A' = \{ (x,y) \in \mathbb{Z}^2 | x > \sqrt{a} \}, B' = \{ (x,y) \in \mathbb{Z}^2 | 0 \le x < \sqrt{a} \}$. First, note that $ x,y \in A'$ iff $ x>y$ and $x,y \in B'$ iff $x<y$ since by positivity of $k$. Rewrite the equation as $$ a=x^2-k(x^2-y^2).$$Claim 1. $A \subseteq B$ Let $(x,y) \in A'$ hence $x>y$. Let $p=x+y, q= x-y$, hence $p>q$ and $p>0$. The equation becomes $$ p^2 -(2-4k)pq+q^2-4a^2=0$$Assume that above equation admits a solution in $(p_0,q_0)$. Then by vieta jump, $(p_0, \frac{q_0^2-4a^2}{p_0})$ is also a solution. But observe that $$ \frac{q_0^2-4a^2}{p_0} < \frac{ q_0^2}{p_0} < q_0<p_0$$We can repeat this prosses and our solution will decrease to negatives eventually. Hence, we will have another solution $(u,v)$ such that $u>0>v$. Then, $x'= \frac{u+v}{2}, y'= \frac{u-v}{2}$. But $x'< y' $. From our first note, $ (x',y') \in B'$. This shows that all $k \in A$ can be shown with $(x,y) \in B'$. By definition of $B$, it means that $k \in B$, which means $A \subseteq B$. Hence we have proven our desired claim. $\square$ Claim 2. $B \subseteq A$. Follows similiarly as Claim 1. $\square$ Hence $A \subseteq B, B \subseteq A$. Then it is well known that $A=B$, therefore we are done!

Substitute $(p, q) = (x+y, x-y)$. Now we can Vieta jump! The given equation rewrites as $$p^2+(2q-4qk)p + q^2-4a = 0.$$This implies that if $(p, q)$ works, then $\left(\frac{q^2-4a}p, q\right)$ works. Claim. For a fixed $k$ for which there exist solutions to the equation, there exists a solution $(p, q)$ with $|p+q|$ minimal such that $0 \leq |x| < \sqrt a$. Proof. It suffices to use the previous transformation. Assume that $x= p+q$ is positive (else, we can flip the solution.) Note that $$\frac{q^2-4a}p + q < p+q$$as $q^2 < p^2$ by definition. On the other hand, $$-p-q < \frac{q^2-4a}p + q \iff (p+q)^2 > 4a,$$which is true as long as $x > \sqrt a$, which means that the transformation yields a smaller solution. Thus when such a process terminates, we are left with a desired solution. $\blacksquare$ Now notice that if $(x, y)$ works, then $(-x, y)$ works too, which solves the problem.

We only need to prove that for $\forall k\in A,k\in B$ and for $\forall k\in B,k\in A.$ For $\forall k\in A,$ let $x,y\in\mathbb Z,x>\sqrt a$ satisfying $k = \frac{x^2-a}{x^2-y^2}.$ Then $ky^2-(k-1)x^2=a.$ $$\Rightarrow (\sqrt ky+\sqrt{k-1}x)(\sqrt ky-\sqrt{k-1}x)=a.$$$$\Rightarrow (\sqrt ky+\sqrt{k-1}x)(\sqrt k-\sqrt{k-1})^2(\sqrt ky-\sqrt{k-1}x)(\sqrt k+\sqrt{k-1})^2=a.$$$$\Rightarrow (\sqrt ky+\sqrt{k-1}x)\left(2k-1-2\sqrt{k(k-1)}\right)(\sqrt ky-\sqrt{k-1}x)\left(2k-1+2\sqrt{k(k-1)}\right)=a.$$$$\Rightarrow\left(\left((2k-1)y-(2k-2)x\right)\sqrt k+\left((2k-1)x-2ky\right)\sqrt{k-1}\right)\left(\left((2k-1)y-(2k-2)x\right)\sqrt k-\left((2k-1)x-2ky\right)\sqrt{k-1}\right)=a.$$$$\Rightarrow ((2k-1)y-(2k-2)x)^2k-((2k-1)x-2ky)^2(k-1)=a.$$Let $X=|(2k-1)y-(2k-2)x|,Y=|(2k-1)x-2ky|.$ Then $kX^2-(k-1)Y^2=a,k=\frac{X^2-a}{X^2-Y^2}.$

Fix $a,k$. Let $x,y$ be nonnegative and set $y=x+d$ so $k(y^2-x^2)=a-x^2$ gives $d$ positive iff $x<\sqrt a$. We get $x^2+2kdx+kd^2-a=0$, so $x=-kd\pm\sqrt{a+k^2d^2-kd^2}$. If $x<\sqrt a$ then $-kd<0$ so $x=-kd+\sqrt{a+k^2d^2-kd^2}$. Then $(x,y)\mapsto(2kd+x,2(k-1)d+y)$ admits and $2kd+x=kd+\sqrt{a+k^2d^2-kd^2}>\sqrt a$. If $x>\sqrt a$ then repeatedly $(x,y)\mapsto(|2kd+x|,|2(k-1)d+y|)$ admits since it decreases $x$ until $d>0$ and $x<\sqrt a$, or until $x\le -kd$. Then $(k-1)x\ge ky,x\ge y$ but $ky^2\ge(k-1)x^2$, impossible

We will in fact prove that if $k$ is an integer, $A=\{k:k\neq 0,1\}=B$ (without using Vieta jumping). First, check that $k=0,1$ is clearly impossible, since in those cases $a=x^2,y^2$, respectively. CASE 1: Let $k>1$, then $a=ky^2-(k-1)x^2$. By setting $x=y-1$, we get $a=k(2y-1)+(y-1)^2>x^2$ for $y>0$, hence we can get every positive $k>1$ in $B$ just by choosing $y>0$ and $a$ not being a perfect square. Suppose $k(2y-1)+(y-1)^2$ is a perfect square for all $y$, that is $t^2=k(2y-1)+(y-1)^2$, $(t+u)^2=k(2y+1)+y^2$, then $u\mid2y+2k-1$, but we can just choose $y$ for $2y+2k-1$ to be prime, hence $u=1,2y+2k-1$, the latter is clearly impossible, therefore $2t+1=2y+2k-1$, that is $t=y+k-1$, yielding $(y-1)^2+k^2+2(y-1)k=(y-1)^2+k(2y-1)\longleftrightarrow k=0,1$. Consequently, all integers $k>1\in B$ CASE 2: Let $k>1, y=x-1$, hence $a=x^2-2kx+k<x^2$ for $x>0$, therefore, we can choose big enough $x$ to be able to choose any $k$. Again, suppose $t^2=x^2-2kx+k$ and using the same argument to get $u=1$, $(t+1)^2=(x+1)^2-2k(x+1)+k$, so $t=x-k$, hence $k=0,1$ again. Consequently, all integers $k>1\in A$ CASE 3: Let $k<0$ and define $0<m=-k$, therefore $a=(m+1)x^2-my^2$ and by setting $y=x-1$, we get $a=x^2+m(2x-1)>x^2$ for $x>0$. By the same argument again, $u=1$, so $t^2=x^2+m(2x-1)$, $(t+1)^2=(x+1)^2+m(2x+1)$, yielding $t=x+m$, but then $(x+m)^2=x^2+m(2x-1)$, so $m=-1,0$. Consequently, all integers $k<0\in B$ CASE 4: Let $k<0$ and define $0<m=-k$, but now set $x=y-1$ to get $a=(y-1)^2-m(2y-1)<(y-1)^2=x^2$ for $y>0$ and can go through all values of $m$ for big enough $y$. Again $u=1$, so $t^2=(y-1)^2-m(2y-1)$, $(t+1)^2=y^2-m(2y+1)$, hence $t=y-m-1$, yielding $(y-1)^2+m^2-2m(y-1)=(y-1)^2-m(2y-1)$, which is equivalent to $m=-1,0$. Consequently, all integers $k<0\in A$