Let $x+y=a$ and $xy=b$ then $a=\lfloor a\cdot \sqrt{a^2-4b}\rfloor$. $1\leq \sqrt{a^2-4b}<1+\frac{1}{a}\implies \frac{a^2-\left(1+\frac{1}{a}\right)^2}{4}<b\leq\frac{a^2-1}{4}$. When $a=1\implies b=0$ contradiction. When $a\geq 2$ , $\frac{a^2}{4}-1<\frac{a^2-\left(1+\frac{1}{a}\right)^2}{4}<b\leq\frac{a^2-1}{4}$ , $b$ is integer. Case1: $a=2n+1\implies b=\frac{a^2-1}{4}\implies x=n+1 , y=n$. Case2: $a=2n\implies n^2-1<b\leq n^2-\frac{1}{4}$ has no integer solution.

A lovely exercise on floor function, it is nice to see that since JBMO 2021 there are many nice algebraic examples like this on junior competitions and TSTs. (Remove inequalities brrrr.) Here is my solution, very similar to @zDerCMO. (There are more straightforward but longer approaches.) Evidently, $x>y$ (otherwise the right-hand side is non-negative). The conditions imply that $x+y = s$ and $xy = p$ are positive integers (where $s\geq 3$ since $s^2 \geq 4p \geq 4$ and $s=2$, $p=1$ would give $x=y=1$) and that $s = \lfloor s \sqrt{s^2 - 4p} \rfloor$. If $s^2 - 4p \geq 2$, then the right-hand side is at least $\lfloor \sqrt{2}s \rfloor > \sqrt{2}s - 1 \geq s$, so $s \leq \sqrt{2} + 1 < 3$, contradiction. But actually $s^2 - 4p > 0$ is an integer (since $s$ and $p$ are integers), hence $s^2 - 4p = 1$, so $x-y = \sqrt{s^2-4p} = 1$. Then $s = 2y+1$ and $p = y(2y+1) = \frac{s(s-1)}{2}$ (which is an integer for every $s$), thus all possible $(x,y)$ are precisely $(\frac{s+1}{2}, \frac{s-1}{2})$, where $s \geq 3$ is an arbitrary odd integer -- actually, this can be rewritten as $(k+1, k)$ for any positive integer $k$.

Very cute problem. We begin by observing that since $x+y = \lfloor x^2-y^2 \rfloor$, then $x+y \in \mathbb{N}$. This implies that $ (x+y)^2 -2xy =x^2+y^2 \in \mathbb{N}$ too. Moreover, we must have: $$ x+y+1 >x^2-y^2 \ge x+y \implies 1+\frac{1}{x+y}> x-y \ge 1$$. Suppose that $x-y>1$ and $x+y \ge 3$. Consequently, we have: \begin{align*} \sqrt{2} >\frac{4}{3} \ge 1+\frac{1}{x+y}> x-y \ge 1\\ \sqrt{2} > x-y >1 \\ 2 >(x-y)^2> 1 \\ -2<2xy-x^2-y^2<-1 \\ x^2+y^2-2 <2xy < x^2+y^2-1 \end{align*}This means that $2xy$ lies between $2$ consecutive integers, which is impossible. We are left to deal with $2$ cases: $x+y=1$, but then $x<1$ and $y<1$, therefore $xy<1$, which is not an positive integer. $x+y=2$, then $x+y=2 \ge 2\sqrt{xy} \implies 1 \ge xy$ with equality case if and only if $x=y=1$. It is easy to check that this is not the solution to the given equation. So in all cases, we have derived the contradiction. This means that we must have $x-y=1$, but then since $x+y$ and $xy$ are integers, we immediately deduce that $(x,y)=(t+1,t)$, where $t$ is some positive integer.

$x+y \in \mathbb{N}$ clearly. Notice that $x+y = \lfloor{x^2-y^2}\rfloor = \lfloor{(x+y)(x-y)}\rfloor$, therefore $x-y \ge 1$, so write $x = 1 + y + c$ with $c \in \mathbb{R}^+$ Notice that $(x+y)^2 - 4xy = (x-y)^2 \in \mathbb{N}$, so $(c+1)^2 = c^2+2c+1 \in \mathbb{N}$. Next up, we want $x+y = 2y+c+1 = \lfloor{x^2-y^2}\rfloor = \lfloor{c^2+2yc+2c+2y+1}\rfloor = \lfloor{(2y+c+1)+(c^2+2yc+c)}\rfloor$. so $2y+c+1 = \lfloor{(2y+c+1)+(c^2+2yc+c)}\rfloor$. Therefore, $c^2+2yc+c \le 1$, or $2yc \le 1-c-c^2$ or $y \le \frac{1-c-c^2}{2c}$ We want $xy \ge 1$ as $xy \in \mathbb{N}_0$. Therefore, $(c+y+1)(y) \ge 1$ or $(c+\frac{1-c-c^2}{2c} + 1)(\frac{1-c-c^2}{2c}) \ge (c+y+1)(y) \ge 1$ $\iff (c^2+c+1)(1-c-c^2) \ge 4c^2$ or $1^2 - (c+c^2)^2 \ge 4c^2$ This doesn't hold if $4c^2 \ge 1$ or if $c \ge \frac{1}{2}$. Notice that $c^2+2c+1 \in \mathbb{N}$ so $c = \sqrt{2}-1$ or $c = 0$ are the only options left to be checked. $c=0$ gives $(x,y) = (t+1,t) \hspace{0.25cm} t \in \mathbb{N}_0$. $c = \sqrt{2}-1$ gives $x+y = 2y + \sqrt{2} \in \mathbb{N}$ and $xy = y^2+y\sqrt{2} \in \mathbb{N}$ Now, write $y = \frac{k-\sqrt{2}}{2}$ with $k \in \mathbb{N}$ so $y^2 + \sqrt{2}y = \frac{k^2-2\sqrt{2}k+2}{4} - \frac{2\sqrt{2}k-4}{4}$ so $y^2+2y = \frac{k^2-2}{4}$ but as $k^2 \equiv 0,1 \pmod{4}$, there is no solution. Therefore $(x,y) = (t+1,t) \hspace{0.25cm} t \in \mathbb{N}$ is the only solution.

It's clear that $x+y$ is an integer. From the equation, $x^2-y^2 \geq x+y$ and $x^2-y^2 \leq x+y+1$. $x^2-y^2 \geq x+y \implies x-y \geq 1$. AM-GM gives $x+y \geq 2\sqrt{xy}$. $xy$ is an integer $\implies$ $x+y \geq 2$. So, $x+y+1 < 2(x+y) \implies x-y < 2$ and $1 \leq x-y < 2$. $(x-y)^2 = (x+y)^2 - 4xy$, which is clearly an integer. Thus $x-y = \sqrt{t}$, where $t$ is some positive integer. Plugging it in the inequality implies $ t = 1, 2$ or $3$. Case $x-y = 1$: $x+y$ is also an integer, hence $x, y$ are integers. This gives us solutions $(x, y) = (d, d-1)$, where d is an integer and $d \geq 2$, which clearly works. Case $x-y = \sqrt{2}$: Let $x+y \equiv v \implies x = \frac{v+\sqrt{2}}{2}$ and $ y = \frac{v-\sqrt{2}}{2}$. Plugging back into the original gives $v = \lfloor \sqrt{2}v \rfloor \implies \sqrt{2}v < v+1 \implies v < \frac{1}{\sqrt{2}-1} = \sqrt{2}+1 < 3$. However, $v = 1$ and $v = 2$ give no solutions. Case $x-y=\sqrt{3}$: We do the same we did with $x-y=\sqrt{2}$. $x+y \equiv v \implies x = \frac{v+\sqrt{3}}{2}$ and $ y = \frac{v-\sqrt{3}}{2}$. $v = \lfloor \sqrt{3}v \rfloor \implies \sqrt{3}v < v+1 \implies v < \frac{1}{\sqrt{3}-1} = \frac{\sqrt{3}+1}{2} < 2$. And $v = 1$ gives no solution. So only solutions are $(x, y) = (d, d-1)$, where $d$ is an integer and $d \geq 2$.

Let $x+y=s \in \mathbb{Z_{+}}$, then $s \leq x^2-y^2=(x+y)(x-y) \leq s+1$, so $1 \leq x-y \leq \frac{s+1}{s} \leq 2$. Now $s^2-(x-y)^2=(x+y)^2-(x-y)^2=4xy$. If $s \vdots 2$, then $x-y=2$ must hold and thus $s=1$ - contradiction. So $s$ is odd. Then if $x-y>1$, $s^2-(x-y)^2$ can not be divisible by $4$. So $x-y=1$, from which $x$ and $y$ are two consective positive integer numbers