There must be $ \left[ {\sqrt {2p} } \right] = \left[ {\sqrt x } \right] + \left[ {\sqrt y } \right]$ .Then try easier.

ISL questions and solutions from 1995-2001 by A.Ajorloo wrote: let $ p = 2n + 1$,first of all note that $ \sqrt {2p} - \sqrt x - \sqrt y\neq 0$,because otherwise we would have $ 2p = x + y + 2\sqrt {xy}$ so if we let $ d = \gcd (x,y)$,using the fact that $ \sqrt {xy}$ is an integer,we would get that $ x = da^2,y = db^2$ so we must have $ 2p = d(a + b)^2$ which is a contradiction. now we have: \begin{eqnarray*}S & = & \sqrt {2p} - \sqrt x - \sqrt y \\ & = & \frac {2p - (\sqrt x + \sqrt y)^2}{\sqrt {2p} + \sqrt x + \sqrt y} \\ & = & \frac {(2p - x - y)^2 - 4xy}{(\sqrt {2p} + \sqrt x + \sqrt y)\left(2p - (\sqrt x - \sqrt y)^2\right)} \\ & = & \frac {(2p - x - y)^2 - 4xy}{(\sqrt {4n + 2} + \sqrt x + \sqrt y)\left(4n + 2 - (\sqrt x - \sqrt y)^2\right)}\end{eqnarray*} now note that the numerator of the last fraction is a natural number,now consider two cases: $ \boxed{\textrm{Case 1. }(2p - x - y)^2 - 4xy\geq 2}$ in this case we'll have: $ S\geq\frac 2{(\sqrt {4n + 2} + \sqrt {4n + 2})(4n + 2)} = \frac 1{\sqrt {4n + 2}(4n + 2)}\geq\frac 1{(6n)^{\frac 32}} > \frac 1{16n^{\frac 32}}$ $ \boxed{\textrm{Case 2. }(2p - x - y)^2 - 4xy = 1}$ in this case let $ 2p - x - y = 2m + 1$ which gives us $ xy = m(m + 1)$,now let: $ \gcd (x,m) = d,\gcd (y,m + 1) = e\Rightarrow x = x_1d,m = m_1d,y = y_1e,m + 1 = m_2e$ which gives us $ x_1y_1 = m_1m_2$. now note that $ \gcd (x_1,m_1) = 1,\gcd (y_1,m_2) = 1$,hence $ x_1 = m_2,y_1 = m_1$,so from $ 2p = x + y + m + (m + 1)$ we get that $ 2p = (d + e)(x_1 + y_1)$ which together with $ x\leq y$ yield that $ x_1 = y_1 = 1$,hence $ x = m,y = m + 1$,but we had $ 2p = x + y + 2m + 1$ so $ m = n$,i.e. $ x = \frac {p - 1}2,y = \frac {p + 1}2$. now we will show that in this case the value of $ S$ is less than the value it had in the previews case: \begin{eqnarray*}S & = & \frac 1{(\sqrt {4n + 2} + \sqrt n + \sqrt {n + 1})(2n + 1 + 2\sqrt {n(n + 1)})} \\ & < & \frac 1{(\sqrt {4n} + \sqrt n + \sqrt n)(2n + 2\sqrt {n^2})} \\ & = & \frac 1{16n^{\frac 32}} \end{eqnarray*} hence the value of $ S$ is minimum whenever $ x = \frac {p - 1}2,y = \frac {p + 1}2$. QED

Solution from Twitch Solves ISL: We claim the best case is if $x = \frac{p-1}{2}$ and $y = \frac{p+1}{2}$. The fact that $\sqrt x + \sqrt y \le \sqrt{2p}$ follows by Jensen. Moreover, it is best possible among choices with $x+y=p$ by Karamata, and hence is also better than any choice with $x+y < p$ (since $(x,y)$ will be worse than $(x,p-y)$). We are left to show that we cannot have \[ \sqrt{\frac{p-1}{2}} + \sqrt{\frac{p+1}{2}} < \sqrt x + \sqrt y < \sqrt{2p} \]for positive integers $x$ and $y$ satisfying $x+y > p$. Let $z = x+y$ and $t = x-y$. Claim: [Calculation] Let $A = z + \sqrt{z^2-t^2}$. Then we have $A < 2p < A + \frac 1A$. Proof. Square both sides: \begin{align*} \sqrt{\frac{p-1}{2}} + \sqrt{\frac{p+1}{2}} &< \sqrt x + \sqrt y < \sqrt{2p} \\ \sqrt{p^2-1} + p &< x+y + 2\sqrt{xy} < 2p \\ \sqrt{p^2-1} &< x+y-p + 2\sqrt{xy} < p. \end{align*}which simplifies to \[ p^2-1 < (x+y-p)^2 + 4xy + 4(x+y-p)\sqrt{xy} < p^2. \]Let $z = x+y$ and $t=x-y$. Simplify further: \begin{align*} p^2-1 &< (z-p)^2 + (z^2-t^2) + 2(z-p)\sqrt{z^2-t^2} < p^2 \\ \iff t^2-1 &< 2(z-p)\left[ z + \sqrt{z^2-t^2} \right] < t^2 \\ \iff (1-t^{-2})(z-\sqrt{z^2-t^2}) &< 2(z-p) < z-\sqrt{z^2-t^2} \\ \iff z + \sqrt{z^2-t^2} &< 2p < z + \sqrt{z^2-t^2} + \frac{1}{z + \sqrt{z^2-t^2}}. \end{align*}Hence done. $\blacksquare$ The idea from here is that ``square roots don't approximate integers too well''. We formalize this using the following lemma. Lemma: Let $n \ge 2$ be an integer which is not a perfect square. Then $n - \sqrt{n^2-1} > \frac{1}{2n}$. Proof. Write $n - \sqrt{n^2-1} = \frac{1}{n + \sqrt{n^2-1}}$. $\blacksquare$ Assume for contradiction $A$ really exists with \[ A < 2p < A + \frac 1A \]and $z > p$. If $A$ is an integer, we have a contradiction. Else, since $z \ge p+1$ it follows that the fractional part of $A$ satisfies \[ 1 - \frac{1}{2(2p-z)} \overset{\text{lemma}}{\ge} \{\sqrt{z^2-t^2}\} = \{A\} > 1 - \frac 1A \]hence \[ A < 2(2p-z) \le 2\left[ 2p-(p+1) \right] = 2p-2 \]which is a contradiction.

The answer is $(x,y)=\left(\tfrac{p+1}{2},\tfrac{p-1}{2}\right)$, which makes $\sqrt{2p}-\sqrt{x}-\sqrt{y}$ positive by Jensen's on the concave function $\sqrt{x}$. Notice that if $\sqrt{2p}-\sqrt{x}-\sqrt{y}$ is positive and minimized, then $$2p-\left(\sqrt{x}+\sqrt{y}\right)^2=2p-x-y-2\sqrt{xy}$$is positive and minimized. Thus, its reciprocal, which is $$\frac{2p-\left(\sqrt{x}-\sqrt{y}\right)^2}{(x-y)^2-4p(x+y-p)},$$is maximized. If the denominator is $1$, then $(x-y)^2 \equiv 1 \pmod{4p}$. Combining this with the bound $x,y<2p$, we find that $|x-y|=1$. If the denominator is at least $2$, then $$\frac{2p-\left(\sqrt{\frac{p+1}{2}}-\sqrt{\frac{p-1}{2}}\right)^2}{1} \geq p \geq \frac{2p-\left(\sqrt{x}-\sqrt{y}\right)^2}{2} \geq \frac{2p-\left(\sqrt{x}-\sqrt{y}\right)^2}{(x-y)^2-4p(x+y-p)},$$which means that we can ignore cases where the denominator is not $1$. Since $|x-y|=1$ and the denominator is $1$, we have $x+y=p$, so we see that $(x,y)=\left(\tfrac{p+1}{2},\tfrac{p-1}{2}\right)$ attains the minimum.

Notice that the $x\le y$ restriction can be immediately relaxed. From here, I claim that $\{x,y\}=\{\lceil 0.5p\rceil ,\lfloor 0.5p\rfloor\}$ is the answer. Claim 1: we cannot have $\sqrt{2p}-\sqrt{x}-\sqrt{y}=0$. Proof: this is equivalent to $y=(\sqrt{2p}-\sqrt{x})^2=2p+x-\sqrt{8px}$. For $\sqrt{8px}$ to be an integer, $x$ is necessarily a multiple of $p$ and $2$, but that forces $-\sqrt{y}$ as nonpositive. Now, to minimize $\sqrt{2p}-\sqrt{x}-\sqrt{y}$ as a positive real, it suffices to make $(\sqrt{x}+\sqrt{y})^2=x+y+\sqrt{4xy}$ as close to $2p$ as possible while being less than it. With the given values it expands to $p+\sqrt{p^2-1}$. Suppose FTSOC that $p+\sqrt{p^2-1}<(\sqrt{a}+\sqrt{b})^2<2p$ for positive integers $a,b$; it's obvious that $a+b>p$. Lemma 1: $f(x)=\sqrt{x}-\sqrt{x-1}$ is strictly decreasing. Proof: differentiate lul Now, suppose that $a+b=p+c$ for some positive integer $c$. Then, $p-c-\sqrt{4ab}$ is necessarily less than $p-\sqrt{p^2-1}$ while being positive. However, by lemma 1 we know that $p-c-\sqrt{(p-c)^2-1}>p-\sqrt{p^2-1}$.

We claim that the minimum value occurs when $x=\frac{p-1}2$ and $y=\frac{p+1}2$. We have \begin{align*} \frac1{\sqrt{2p}-\sqrt{\frac{p-1}2}-\sqrt {\frac{p+1}2}}&=\frac{\sqrt{2p}+\sqrt{\frac{p+1}2}+\sqrt{\frac{p-1}2}}{p-\sqrt{p^2-1}}\\ &=\left(\sqrt{2p}+\sqrt{\frac{p+1}2}+\sqrt{\frac{p-1}2}\right)(p+\sqrt{p^2-1})\\ &>4\sqrt{2p-2}\sqrt{p^2-1}\\ &=4(p-1)\sqrt{2p+2}\\ &>4(p-1)\sqrt{2p} \end{align*} Now, suppose that there exists $x$ and $y$ such that $\frac1{\sqrt{2p}-\sqrt x-\sqrt y}>4(p-1)\sqrt{2p}$. Then, we have \begin{align*} 4(p-1)\sqrt{2p}&<\frac1{\sqrt{2p}-\sqrt x-\sqrt y}\\ &=\frac{\sqrt{2p}+\sqrt x+\sqrt y}{2p-x-y-2\sqrt{xy}}\\ &<\frac{2\sqrt{2p}}{2p-x-y-2\sqrt{xy}}\\ 2(p-1)&<\frac1{2p-x-y-2\sqrt{xy}}. \end{align*} Now, let $n=2p-x-y$. We have $\frac1{n-\sqrt{4xy}}=\frac{n+\sqrt{4xy}}{n^2-4xy}$, so if $n^2-4xy\geq2$, then we have $$2(p-1)<\frac{2p-x-y+2\sqrt{xy}}2.$$However, this is a contradiction since $2p-(\sqrt x-\sqrt y)^2<2p$. Therefore, we must have $n^2-4xy=1$. This means that $4p^2-4p(x+y)+(x-y)^2=1$. Since $2p>x+y+2\sqrt{xy}>x+y$, this means that $1=4p^2-4p(x+y)+(x-y)^2>-4p^2+(x-y)^2$, this means that $4p^2+1>(x-y)^2$, so $|x-y|\leq2p$. Since $(x-y)^2\equiv1\pmod p$, we must have $y-x=1,p-1,p+1,2p-1$. Taking mod $2$, we also have that $x-y$ is odd, which means that $y-x=1,2p-1$. Case 1: $y-x=2p-1$ Then, $4p^2-4p(x+y)+4p^2-4p+1=1$, so $p-(x+y)+p-1=0$. This implies that $x+y=2p-1$, so $2p>x+y+2\sqrt{xy}=2p-1+2\sqrt{xy}$, which implies $1>2\sqrt{xy}$, which is impossible. Case 2: $y-x=1$ Then, $4p^2-4p(x+y)+1=1$, which means that $x+y=p$. This means that $x=\frac{p-1}2$ and $y=\frac{p+1}2$. Therefore, the positive integers $x$ and $y$ such that $\sqrt{2p}-\sqrt x-\sqrt y$ is minimal is $\boxed{(x,y)=\left(\frac{p-1}2,\frac{p+1}2\right)}$.

Does this work? It seems an awful lot simpler than all of the other solutions. The answer is $(x, y) = \left(\frac{p-1}2, \frac{p+1}2\right)$. By Jensen/Karamata, this pair is maximal for all $x + y \leq p$. Now assume for the sake of contradiction that there exists $x+y = p + k$ with $k \geq 1$ such that $$\sqrt{\frac{p-1}2} + \sqrt{\frac{p+1}2} < \sqrt x + \sqrt y \leq \sqrt{2p}.$$Squaring and simplifying, $$\sqrt{p^2 - 1} < k +2\sqrt{xy} \leq p.$$Notice that we have $$\sqrt{(p-k)^2 - 1} < \sqrt{p^2 - 1} - k < p-k$$because $$p - \sqrt{p^2 - 1} = \frac 1{p+\sqrt{p^2-1}} < \frac 1{p-k+\sqrt{(p-k)^2 - 1}} = (p-k)-\sqrt{(p-k)^2 - 1}.$$Hence $(2\sqrt{xy})^2$ cannot be an integer, which is an obvious contradiction. [Addendum: Technically we should also address the possible case $\sqrt x + \sqrt y = \sqrt{2p}$. Fortunately, this follows because $2p$ is squarefree and square roots have no nontrivial linear relationships.]

The answer is $(x,y) = ((p-1)/2, (p+1)/2)$. This can be checked to satisfy $\sqrt{2p} - \sqrt{x} - \sqrt{y} \ge 0$. Now suppose there were $x,y$ such that that value was still nonnegative and even smaller. Let $t = (\sqrt{x} + \sqrt{y})^2 = x + y + \sqrt{4xy}$ and $n = \lceil \sqrt{4xy} \rceil$. We wish to minimize $2p-t$. Claim: $\sqrt{2p} - \sqrt{x} - \sqrt{y}\ne 0$. Proof: Suppose otherwise. Then $2p + x - 2\sqrt{2px}= y$, which implies that $2px$ is a perfect square, so $p\mid x$. Clearly $x<2p$, so $x=p$. But then $2p^2$ is not a perfect square, contradiction. $\square$ Claim: $t> p + \sqrt{p^2 -1 }$ Proof: We have \[\sqrt{(p-1)/2} + \sqrt{(p+1)/2} < \sqrt{x} + \sqrt{y} \le \sqrt{2p}\]Squaring both sides gives the desired result.$\square$ Claim: $x+y>p$ Proof: If $x+y = p$, then $x = (p-1)/2, y = (p+1)/2$ gives the maximal possible value of $xy$. If $x+y<p$, then $t\le 2(x+y) \le 2p-2 \le p + \sqrt{p^2 - 1}$ by AM-GM, contradiction. $\square$ Thus, we have $2p-t<p - \sqrt{p^2 - 1}$. Now, we also have $n^2 \ge 4xy + 1$, $n\le 2p - x - y < p$ because $x+y+\sqrt{4xy} = t<2p$. Claim: $f(x) = x - \sqrt{x^2 - 1}$ is strictly decreasing over the interval $(1,\infty)$. Proof: Suppose not and there existed $a>b>1$ with \[a - \sqrt{a^2 - 1} > b- \sqrt{b^2 - 1}\]Then taking the reciprocal of both sides gives that \[a + \sqrt{a^2 - 1} < b + \sqrt{b^2 - 1},\]which is absurd. $\square$ So \[2p-t \ge 2p-x - y - \sqrt{4xy} \ge n - \sqrt{n^2 -1} \ge p - \sqrt{p^2 - 1},\]contradiction because we already have $2p-t<p - \sqrt{p^2 - 1}$.

The answer is $(x,y) = ((p-1)/2, (p+1)/2)$. This can be checked to satisfy $\sqrt{2p} - \sqrt{x} - \sqrt{y} \ge 0$. Now suppose there were $x,y$ such that that value was still nonnegative and even smaller. Let $t = (\sqrt{x} + \sqrt{y})^2 = x + y + \sqrt{4xy}$ and $n = \lceil \sqrt{4xy} \rceil$. We wish to minimize $2p-t$. Claim: $\sqrt{2p} - \sqrt{x} - \sqrt{y}\ne 0$. Proof: Suppose otherwise. Then $2p + x - 2\sqrt{2px}= y$, which implies that $2px$ is a perfect square, so $p\mid x$. Clearly $x<2p$, so $x=p$. But then $2p^2$ is not a perfect square, contradiction. $\square$ Claim: $t> p + \sqrt{p^2 -1 }$ Proof: We have \[\sqrt{(p-1)/2} + \sqrt{(p+1)/2} < \sqrt{x} + \sqrt{y} \le \sqrt{2p}\]Squaring both sides gives the desired result.$\square$ Claim: $x+y>p$ Proof: If $x+y = p$, then $x = (p-1)/2, y = (p+1)/2$ gives the maximal possible value of $xy$. If $x+y<p$, then $t\le 2(x+y) \le 2p-2 \le p + \sqrt{p^2 - 1}$ by AM-GM, contradiction. $\square$ Thus, we have $2p-t<p - \sqrt{p^2 - 1}$. Now, we also have $n^2 \ge 4xy + 1$, $n\le 2p - x - y < p$ because $x+y+\sqrt{4xy} = t<2p$. Claim: $f(x) = x - \sqrt{x^2 - 1}$ is strictly decreasing over the interval $(1,\infty)$. Proof: Suppose not and there existed $a>b>1$ with \[a - \sqrt{a^2 - 1} > b- \sqrt{b^2 - 1}\]Then taking the reciprocal of both sides gives that \[a + \sqrt{a^2 - 1} < b + \sqrt{b^2 - 1},\]which is absurd. $\square$ So \[2p-t \ge 2p-x - y - \sqrt{4xy} \ge n - \sqrt{n^2 -1} \ge p - \sqrt{p^2 - 1},\]contradiction because we already have $2p-t<p - \sqrt{p^2 - 1}$.

The answer is $x,y=\frac{p-1}2,\frac{p+1}2.$ This achieves $2p-(\sqrt x+\sqrt y)^2=p-\sqrt{p^2-1}.$ If $x+y\ge p$ then we want $(\sqrt x+\sqrt y)^2=x+y+\sqrt{4xy}$ to be as close to $2p$ as possible. We see it cannot equal $2p$ because $xy$ needs to be a square and then we may set $x=ab^2,y=ac^2$ and we reach a contradiction as $2p$ is squarefree. Thus we want $\sqrt{4xy}$ to be less than $2p-x-y=n.$ We must have $4xy\le n^2-1,$ so $2p-(\sqrt x+\sqrt y)^2\ge n-\sqrt{n^2-1}.$ We have $n\le p,$ and it is easy to see that $x-\sqrt{x^2-1}$ is increasing, so $n-\sqrt{n^2-1}\le p-\sqrt{p^2-1}.$ We can check that equality only holds in the aforementioned $x,y.$ If $x+y<p$ then $(\sqrt x+\sqrt y)^2=x+y+2\sqrt{xy}\le 2(x+y)\le 2p-2<p+\sqrt{p^2-1}$ by AM-GM.

Here is a sketch The answer is $(x,y) = ((p-1)/2, (p+1)/2)$, We have that $$2p-(\sqrt x+\sqrt y)^2=p-\sqrt{p^2-1}.$$ Now assume $x+y< p$ then you have that. \[(\sqrt{x} + \sqrt{y})^2<\sqrt{p^2-1}+p \]By using AM-GM. Now for case 2: $x+y \geq p$ We need $\sqrt{4xy} < 2p-x-y$, this force $4xy \leq n^2-1$ We can easily see that the function $f(x)=x– \sqrt{x^2-1}$ is increasing, we see that we can only get equality if we have $x,y$ as above.