$(a,b)=(9,9)$. straightforward algebra

Of course straightforward algebra work. Here's a solution to avoid all the unnecessary bash. Write $\sqrt{a - 5} = x, \sqrt{b - 5} = y$. Then we want to solve \begin{align*} \sqrt{x^2 + 5} + \sqrt{y^2 + 5} &= 6 \\ x + y &= 4 \end{align*}But to solve this, we just note that \begin{align*} 36 &= (\sqrt{x^2 + 5} + \sqrt{y^2 + 5})^2 \\ &= (x^2 + 5) + (y^2 + 5) + 2 \sqrt{(x^2 + 5)(y^2 + 5)} \\ &\stackrel{\text{CS}}{\ge} (x^2 + 5) + (y^2 + 5) + 2(xy + 5) = (x + y)^2 + 20 = 36 \end{align*}and as equality hold, we must have $x = y = 2$ and thus $a = b = 9$ as desired.

Use Minkowski's Inequality. Problem is equivalent to $$\sqrt{x+5}+\sqrt{y+5}=6$$$$\sqrt{x}+\sqrt{y}=4$$where $a-5=x$ and $b-5=y$. But via Minkowski's Inequality $$\sqrt{x+5}+\sqrt{y+5}\overbrace{\geq}^{Minkowski} \sqrt{(\sqrt{x}+\sqrt{y})^2+20}=\sqrt{16+20}=6$$which implies $a=b$. The result follows.

What if we want to bash?

Bruh I didn't manage to answer this on the test