Let O, R be the triangle circumcenter and circumradius. Let P be any point (not necessarily on the incircle) and denote r = OP (not necessarily the inradius). WLOG, let P be in the convex plane sector bounded by the rays (OA, (OB. Denote $\angle AOP = \phi.$ Then $\angle BOP = 120^\circ-\phi$ and $\angle COP = 120^\circ+\phi.$ Summing cosine laws for the $\triangle AOP,\ \triangle BOP,\ \triangle COP,$ $AP^{2}+BP^{2}+CP^{2}=$ $= 3(R^{2}+r^{2})-2Rr (\cos \phi+\cos (120^\circ-\phi)+\cos (120^\circ+\phi)) = 3(R^{2}+r^{2})$ because $\cos \phi+\cos (120^\circ-\phi)+\cos (120^\circ+\phi) = \cos \phi+2 \cos \phi \cos 120^\circ = 0$ If $P \in \omega,$ $r = \frac{R}{2}$ is the inradius, and $AP^{2}+BP^{2}+CP^{2}= \frac{15}{4}R^{2}= \frac{5}{4}AB^{2}= 5.$ Assume P is not on the circumcircle (O), which is certainly true for $P \in \omega.$ Then APBC is not cyclic and by Ptolemy inequality for this quadrilateral, $CP \cdot AB < AP \cdot BC+BP \cdot CA,\ \ \ CP < AP+BP$ $BP \cdot CA < CP \cdot AB+AP \cdot BC,\ \ \ BP < CP+AP$ $AP \cdot BC < BP \cdot CA+CP \cdot AB,\ \ \ AP < BP+CP$ Thus the segments a = AP, b = BP, c = CP can form a triangle. Let (P) be a circle centered at P and tangent to (O), i.e., with radius $p = R-OP = R-r.$ Let $t_{A}= \sqrt{a^{2}-p^{2}},$ $t_{B}= \sqrt{b^{2}-p^{2}},$ $t_{C}= \sqrt{c^{2}-p^{2}}$ be tangent lengths from A, B, C, to (P). By Purser's theorem, $t_{A}\cdot BC+t_{B}\cdot CA = t_{C}\cdot AB,\ \ \ t_{A}+t_{B}= t_{C},\ \ \ \sqrt{a^{2}-p^{2}}+\sqrt{b^{2}-p^{2}}= \sqrt{c^{2}-p^{2}}$ Squaring this 2x to get rid of square roots, $(a^{2}+b^{2}-c^{2})^{2}-2p^{2}(a^{2}+b^{2}-c^{2})+p^{4}= 4(a^{2}-p^{2})(b^{2}-p^{2})$ Denote area of the $\triangle (abc)$ as $\triangle.$ Substituting $(a^{2}+b^{2}-c^{2})^{2}= 4a^{2}b^{2}-16 \triangle^{2}$ (cosine law and area formula) and $a^{2}+b^{2}+c^{2}= 3(R^{2}+r^{2}),$ $16 \triangle^{2}= 2p^{2}(a^{2}+b^{2}+c^{2})-3p^{4}= 6p^{2}(R^{2}+r^{2})-3p^{4}=$ $= 6(R-r)^{2}(R^{2}+r^{2})-3(R-r)^{4}= 3(R-r)^{2}(R+r)^{2}$ $\triangle = \frac{\sqrt{3}}{4}(R^{2}-r^{2})$ If $P \in \omega,$ $r = \frac{R}{2}$ is the inradius, $R^{2}-r^{2}= \frac{3}{4}R^{2}= \frac{AB^{2}}{4}= 1$ and $\triangle = \frac{\sqrt{3}}{4}.$

a) Let $A=(2,\sqrt3), B=(2,0),C=(0,0).$ Then $I,$ the incenter, is $(1,\frac{\sqrt3}{3}).$ The equation of the incircle is $(x-1)^{2}+(y-\frac{\sqrt3}{3})^{2}=\frac{1}{3}$ $\iff 3x^{2}+3y^{2}-6x-2\sqrt3 y=-3.$ If $p=(x,y),$ $AP^{2}+BP^{2}+CP^{2}=(x-1)^{2}+(y-\sqrt3)^{2}+(x-2)^{2}+y^{2}+x^{2}+y^{2}=3x^{2}+3y^{2}-6x-2\sqrt3 y+8=5.$ b)Rotate triangle $ABC$ around $B$ by $\pi/3,$ so that $C$ coincides with $A$ and $P$ goes to $P'.$ Since $BP=BP'$ and $\angle PBP'=\pi/3,$ $BPP'$ is equilateral and $BP=PP'.$ Also, $CP=AP',$ so triangle $APP'$ has side lengths $AP,BP,CP.$ Let $[ABC]$ denote the area of triangle $ABC.$ Let the area of the triangle with sidelengths $AP,BP,CP$ be $K$ Then $[APP']+[BPP']=K+BP^{2}\frac{\sqrt3}{4}=[APB]+[AP'B]=[APB]+[CPB]$ Similarly, $K+AP^{2}\frac{\sqrt3}{4}=[APB]+[APC]$ $K+CP^{2}\frac{\sqrt3}{4}=[APC]+[BPC]$ Adding the 3 equations, $3K+(AP^{2}+BP^{2}+CP^{2})\frac{\sqrt3}{4}=3K+\frac{5\sqrt3}{4}=2([APC]+[BPC]+[APB])=2[ABC]=2\sqrt3$ $\iff K=\frac{\sqrt3}{4}$

http://www.cut-the-knot.org/wiki-math/index.php?n=MathematicalOlympiads.IbAMO1992Problem3 Vo Duc Dien