Please post your solutions. Use $\LaTeX$ please! Please omit jokes/smilies etc. Comments and generalizations are welcome. If you noticed that the comment has been discussed elsewhere, please provide a link. If you don't know the link(s) do NOT post and state that the problems has been discussed many times. If the provided solution is not complete, right or in $\LaTeX$-style I would be happy if you could (re-) post your/the solution in this thread again! At the end of your (re-)written solution post the links to those insufficient solutions as follows:

Remark

Happy Problem-Solving!

orl wrote: Let $\triangle XOY = \frac{\pi}{2}$ Is this?

I think you mean $\angle XOY$ and $\angle XOP$, instead of $\triangle XOY$ and $\triangle XOP$.

Does anybody know how to solve this problem? It seems to have a very ugly answer based on calculus.

Lemma 1: The inradius of $\triangle MON$ is $\frac{1}{2}(OM+ON-MN)$. Proof: Let the incircle centered at $I$ meet $MO,ON,MN$ at $A,B,C$ respectively. Note that $AOBI$ is a square of sidelength $r$. Also note $MA=MC$ and $NB=NC$, hence $OM+ON-MN=MA+OA+NB+OB-MC-NC=OA+OB=2r$. The lemma follows. $\square$ From lemma 1 it suffices to maximize $2r$. Denote the line through $P$ as $\ell_P$. We use Cartesian Coordinates. Set $P=\left (\frac{1}{2},\frac{\sqrt{3}}{2}\right )$, and $I=(r,r)$. Lemma 2: We must have $r\leq\sqrt{3}$. Proof: Recall that $I$ must be inside $\triangle MON$. Let the point where $\ell_P$ intersects the line $y=x$ be $X$. clearly the furthest this point can be from the origin is $\sqrt{6}$ giving $r\leq\sqrt{3}$. $\square$ Let $d(A,B)$ denote the shortest distance between a point on $A$ and a point on $B$. Now from definition of shortest distance between a point and a line we have $r=d(I,\ell_P)\leq d(I,P)$. Now we have \begin{align*} d(I,P)&\geq r \\ \left(r-\frac{1}{2}\right)^2+\left (r-\frac{\sqrt{3}}{2}\right )^2&\geq r^2 \\ (2r-1)^2+(2r-\sqrt{3})^2-(2r)^2 &\geq 0 \end{align*} Solving the quadratic inequality gives $2r\geq 1+\sqrt{3}+\sqrt[4]{12}\geq 4$ or $2r\leq 1+\sqrt{3}-\sqrt[4]{12}$. We have shown that $r\leq \sqrt{3}<2$ so the first case is impossible and thus the maximum value is $\boxed{1+\sqrt{3}-\sqrt[4]{12}}$ which is easily obtained through construction.

It is possible to achieve the quadratic without the use of coordinates in a slightly different manner. In particular define points as above. The $IP^2=IO^2+OP^2-2(IO)(OP)cos(15^\circ)$ or using that $IP\le r$ then $r^2\le1+2r^2-2\sqrt{2}rcos(15^\circ)$ and the above follows. However an easier way to prove that $r\le\frac{\sqrt{3}}{2}$ is to let $R$ be the projection of $I$ on $OX$ and $Q$ be the projection of $I$ onto $OY$. Suppose that $r>\frac{\sqrt{3}}{2}$, then $P$ is contained inside $ORIQ$, which is obviously a contradiction.