$(x + y)^2 - xy = (a - z)^2 - z^2 = a^2 - 2az = b^2$ $\Longrightarrow$ $z = \frac{a^2 - b^2}{2a}$ $x + y = a - z$ and $xy = z^2$ Solving the system we get $t_{1/2} = \frac{a^2 + b^2 \pm \sqrt{(3b^2 - a^2)(3a^2 - b^2)}}{4a}$. If $x, y, z > 0$ then $a > 0$ and $a^2 - b^2 > 0$ $\Rightarrow$ $a > \left|b\right|$. For the existance of the root and for getting distinct numbers $(3b^2 - a^2)(3a^2 - b^2) > 0$ $\Rightarrow$ $3b^2 > a^2$ and finally $\sqrt{3}\left|b\right| > a$.

Andreas wrote: $(x + y)^2 - xy = (a - z)^2 - z^2 = a^2 - 2az = b^2$ $\Longrightarrow$ $z = \frac{a^2 - b^2}{2a}$ $x + y = a - z$ and $xy = z^2$ Solving the system we get $t_{1/2} = \frac{a^2 + b^2 \pm \sqrt{(3b^2 - a^2)(3a^2 - b^2)}}{4a}$. If $x, y, z > 0$ then $a > 0$ and $a^2 - b^2 > 0$ $\Rightarrow$ $a > \left|b\right|$. For the existance of the root and for getting distinct numbers $(3b^2 - a^2)(3a^2 - b^2) > 0$ $\Rightarrow$ $3b^2 > a^2$ and finally $\sqrt{3}\left|b\right| > a$. Why $a^2 - 2az = b^2$ :

Sorry is very nice

only from $3(x^2+y^2+z^2)\geq(x+y+z)^2$ we get $ \sqrt{3}\left|b\right| > a $, but i didn't use condition $xy=z^2$.

I'm sure there's an easier way, anyone have it?

For $a=0$, we get $(x,y,z)=(0,0,0)$. For $a \neq 0$, we get $(x,y,z) \in \{(s_1,s_2, z_0),(s_2, s_1, z_0)\}$, where $$z_0=\frac{a^2-b^2}{2a} \text{ and } s_{1,2}= \frac{a^2+b^2±\sqrt{(3a^2-b^2)(3b^2-a^2)}}{4a}$$the answers are positive and distinct iff $3b^2>a^2>b^2$ & $a>0$.

Substitute $z=\sqrt{xy}$ in all instances. Let $x+y=p$ and $\sqrt{xy}=q.$ The first equation simplifies to $$(x+y)+\sqrt{xy}=p+q=a.$$Note that $x^2+y^2=p^2-2q^2,$ so the second equation simplifies to $$(x^2+y^2)+(\sqrt{xy})^2=p^2-q^2=(p+q)(p-q)=a(p-q)=b^2.$$Note that by AM-GM, $p \geq 2q$ thus $p-q \neq 0$ and $b=0$ iff $a=0,$ in which case the trivial inequality on the second equation yields that the only solution is $(x,y,z)=(0,0,0),$ which is not possible since $x,y,z$ must be distinct. Thus, $a,b \neq 0$ and we may obtain $$p-q=\frac{b^2}{a}$$$$xy=q^2=\left(p-\frac{b^2}{a}\right )^2\qquad (1)$$Now, manipulate the first equation to obtain $$q=z=a-p$$$$q^2=(p-a)^2=p^2-2ap+a^2,$$but since the second equation gives $p^2=b^2+q^2,$ we have $$q^2=b^2+q^2-2ap+a^2$$$$x+y=p=\frac{a^2+b^2}{2a}.$$Plugging this into $(1)$ yields that (with simplification) $$xy=\frac{(a^2-b^2)^2}{4a^2}.$$We use Vieta's by setting a quadratic $f(t)$ whose roots $x,y$ satisfy these conditions $$f(t)=t^2-\frac{a^2+b^2}{2a}t+\frac{(a^2-b^2)^2}{4a^2}.$$Using the quadratic formula and simplifying yields $$x,y=\frac{a^2+b^2 \pm \sqrt{(3a^2-b^2)(3b^2-a^2)}}{4a}$$Note that if $x,y$ exist and are distinct positive real numbers, then $z=\sqrt{xy}$ must also exist and be distinct (not trivial, but easy to prove by contradiction by setting $z=x,y,$ which results in $x=y,$ a contradiction). So, it is necessary that (for distinctness and existence of solutions) $$(3a^2-b^2)(3b^2-a^2)>0 \implies 3b^2>a^2,3a^2>b^2.$$Since $a=x+y+z>0,$ this is equivalent to $\sqrt{3} |b|>a.$ Then, we must also have $$a^2 + b^2 >sqrt{(3b^2 - a^2)(3a^2 - b^2)}$$$$(a^2+b^2)^2>(3b^2 - a^2)(3a^2 - b^2)$$$$(a^2-b^2)^2>0,$$which is clearly true as long as $a,b$ are real. So, $\sqrt{3} |b|>a$ for reals $a,b$ is both a necessary and sufficient condition for $x,y,$ and since $z=\frac{a^2-b^2}{2a},$ we must have that $a^2 > b^2$ or $a>|b|,$ which replaces our previous bound. Thus, our long answer: (paying no attention to which is assigned to $x$ or $y$) $$x,y=\frac{a^2+b^2 \pm \sqrt{(3a^2-b^2)(3b^2-a^2)}}{4a},z=\frac{a^2-b^2}{2a}\qquad \forall \qquad |b|<a<\sqrt{3}|b|.$$