You can get the estimation for the area from Ptolemey's inequality. Try to build some triangles up to parallelograms to get convenient quadraliteral.

I think this was used in some Romanian TST (1999, if I am not mistaken). Anyway, the main idea is that the maximal area is achieved when $O$ is the orthocenter of $ABC$. After that, it does become rather ugly, from what I can remember.

According to the problem Triangle inequality, for an arbitrary point $O$ inside of a triangle $\triangle ABC$, the following inequality holds (with equality when $O$ is the triangle orthocenter): $OA \cdot OB \cdot BC + OB \cdot OC \cdot CA + OC \cdot OA \cdot AB \ge AB \cdot BC \cdot CA$ Let $a = BC, b = CA, c = AB$, $R, R_A, R_B, R_C$ the circumradii of the triangles $\triangle ABC, \triangle OBC, \triangle OCA, \triangle OAB$. Using the formula $|\triangle ABC| = \frac{abc}{4R}$, etc, the inequality can be written as $OA \cdot OB \cdot a + OB \cdot OC \cdot b + OC \cdot OA \cdot c \ge abc$ $R \cdot |\triangle ABC| \le R_A \cdot |\triangle OBC| + R_B \cdot |\triangle OCA| + R_C \cdot |\triangle OAB|$ $|\triangle ABC| \le \frac{R_A}{R} \cdot |\triangle OBC| + \frac{R_B}{R} \cdot |\triangle OCA| + \frac{R_C}{R} \cdot |\triangle OAB|$ The area $|\triangle ABC|$ is maximum at equality, i.e., when $O$ is the orthocenter. If this is the case, then $R_A = R_B = R_C = R$ and the inequality turns into a trivial equality: $|\triangle ABC| = |\triangle OBC| + |\triangle OCA| + |\triangle OAB|$ Thus the problem is to find the triangle area given the distances of the orthocenter $O$ from the vertices $A, B, C$. Let $\alpha, \beta, \gamma$ be the angles at the vertices $A, B, C$. The distances $OA, OB, OC$ are equal to $OA = 2R \cos \alpha,\ \ OB = 2R \cos \beta,\ \ OC = 2R \cos \gamma$ $\frac{OA}{OB} = \frac{\cos \alpha}{\cos \beta} = \frac{4}{2 \sqrt 3} = \frac{2}{\sqrt 33},\ \ \frac{\cos^2{\alpha}}{\cos^2{\beta}} = \frac 4 3$ $\frac{OC}{OA}= \frac{\cos \gamma}{\cos \alpha} = \frac{\sqrt{22}}{4} = \frac{\sqrt{11}}{2 \sqrt 2},\ \ \frac{\cos^2{\gamma}}{\cos^2{\alpha}} = \frac{11}{8}$ Since $\alpha + \beta + \gamma = \pi$, $\cos \gamma = -\cos (\alpha + \beta) = \cos \alpha \cos \beta - \sin \alpha \sin \beta$. $\frac{\cos^2{\alpha} \cos^2{\beta} - 2 \cos \alpha \cos \beta \sin \alpha \sin \beta + \sin^2{\alpha} \sin^2{\beta}}{\cos^2{\alpha}} = \frac{11}{8}$ Substituting $\sin \alpha = \sqrt{1 - \cos^2{\alpha}},\ \sin \beta = \sqrt{1 - \cos^2{\beta}}$, denoting $x = \cos^2{\alpha}$ and substituting $\cos^2{\beta} = \frac 3 4\ \cos^2{\alpha} = \frac{3x}{4}$, this becomes $\frac{3x^2}{4} - x \sqrt{3(1 - x)\left(1 - \frac{3x}{4}\right)} + (1 - x)\left(1 - \frac{3x}{4}\right) = \frac{11x}{8}$ $12x^2 - 25x + 8 = 4x \sqrt{3(3x^2 - 7x + 4)}$ $144x^4 + 625x^2 + 64 - 600x^3 + 192x^2 - 400x = 144x^4 - 336x^3 + 192x^2$ $264x^3 - 625x^2 + 400x - 64 = 0$ Thus we obtained a cubic equation, as expected (see the problem Triangle edges and orthocenter). We can test this cubic equation for rational roots with the help of the rational root theorem: Possible rational roots are $\frac p q$, where $p$ divides $64 = 2^6$ and $q$ divides $264 = 2^3 \cdot 3 \cdot 11$. Or we can solve numerically and factor out any rational roots we spot. The above equation has a rational root $x = \frac{8}{33}$. Factoring it out yields $264x^3 - 625x^2 + 400x - 64 = (33x - 8)(8x^2 - 17x + 8) = 0$ and the remaining 2 roots are $x = \frac{17 \pm \sqrt{33}}{16}$. The root with the plus sign is greater than 1, i.e., not acceptable for a cosine. The remaining irrational root is also not acceptable, because for $x = \frac{17 - \sqrt{33}}{16}$, $12x^2 - 25x + 8 = \frac{12(289 + 33 - 34 \sqrt{33})}{256} - \frac{25(17 - \sqrt{33})}{16} + 8 =$ $= \frac{483 - 850 + 256 - (51 - 50) \sqrt{33}}{32} = \frac{\sqrt{33} - 111}{32} < 0$ i.e., it is not a root of the original equation before we squared it. Consequently, it must be $x = \cos^2{\alpha} = \frac{8}{33}$, $\cos \alpha = \pm \frac{2 \sqrt 2}{\sqrt{33}}$. The minus sign is not acceptable, because the angle $\alpha$ would be obtuse, the orthocenter outside of it, and it obviously would not have maximum area for the given lengths $OA, OB, OC$. Therefore, $\cos \alpha = \frac{2 \sqrt 2}{\sqrt{33}},\ \ \cos \beta = \frac{\sqrt 3}{2} \cos \alpha = \frac{\sqrt 2}{\sqrt{11}},\ \ \cos \gamma = \frac{\sqrt{11}}{2 \sqrt 2} \cos \alpha = \frac{1}{2 \sqrt 3}$ $\sin \alpha = \frac{5}{\sqrt{33}},\ \ \sin \beta = \frac{3}{\sqrt{11}},\ \ \sin \gamma = \frac{\sqrt{11}}{2 \sqrt 3}$ $R = \frac{OA}{2 \cos \alpha} = \frac{OB}{2 \cos \beta} = \frac{OC}{2 \cos \gamma} = \sqrt{\frac{33}{2}}$ $a = \frac{2R}{\sin \alpha} = \frac{33 \sqrt 2}{5},\ \ b = \frac{2R}{\sin \beta} = 11 \sqrt 2,\ \ c = \frac{2R}{\sin \gamma} = 6 \sqrt 2$ $|\triangle ABC| = \frac 1 2\ bc \sin \alpha = 10 \sqrt{33}$

I saw this problem back then, and liked it- I've proposed versions of it myself. The answer requires the solution of a cubic equation, most conveniently for $2R$. If $OA=a,OB=b,OC=c$, $(2R)^3-(a^2+b^2+c^2)2R-2abc=0$ Then the area is $\frac{ab\sqrt{4R^2-c^2}}{4R}+\frac{ac\sqrt{4R^2-b^2}}{4R}+ \frac{bc\sqrt{4R^2-a^2}}{4R}$. In this case, that cubic is $(2R)^3-(16+12+22)(2R)-16\sqrt{66}=0$, or $2R^3-25R-4\sqrt{66}=0$. There aren't many possible reasonable roots, and we find that the unique positive root is $R=\sqrt{\frac{33}{2}}.$ It's trivial to show that the maximum occurs when $O$ is the orthocenter- look at what happens if you fix two vertices and let the other vary. It's nice to rig it backward from a nice triangle: I like $a=25,b=33,c=39$.

.

yetti wrote: $|\triangle ABC| \le \frac{R_A}{R} \cdot |\triangle OBC| + \frac{R_B}{R} \cdot |\triangle OCA| + \frac{R_C}{R} \cdot |\triangle OAB|$ The area $|\triangle ABC|$ is maximum at equality, i.e., when $O$ is the orthocenter. I don't quite see this. It's true that there is equality when $O$ is the orthocentre, but the right-hand side is not constant, as far as I can see. So, you can't conclude that the area is maximal simply when there is equality.

$O$ must be the orthocenter in order to have a critical point of the oriented area function. For $A,B$ fixed and $C$ variable, the tangent line at $C$ must be parallel to $AB$ at a critical point $(A,B,C)$, which means $OC$ is perpendicular to $AB$. If one point $A$ is fixed, the critical points come in pairs under reflection in $OA$, which exchanges max/min. [(Edit: <deleted some stupid remarks about number and type of critical points>]. So my guess is that we want the case when $O$ is orthocenter and it is inside $ABC$, and this is the unique maximum of unoriented area.

So much effort on the trivially easy part! Fix $B$ and $C$; the maximum area clearly occurs when $A$ is as far away from $BC$ as possible, which occurs when $OA$ is perpendicular to $BC$ and $A$ is farther from $BC$ than $O$. Therefore, if a maximum exists, it must come when $O$ is the orthocenter and $O$ is inside $ABC$. The configuration can be specified by two angles $\alpha=AOB$ and $\beta=BOC$, with compact domain $\alpha\ge0$, $\beta\ge0$, $\alpha+\beta\le2\pi$. The area is the continuous function $\frac12|ab\sin\alpha+bc\sin\beta+ac\sin(2\pi-\alpha-\beta)|$, which must take its maximum by the Extrame Value theorem. Actually, any exterior orthocenter possibilities would be saddle points.

Thanks, that remark about the saddle points makes it clear (and the argument for $O$ being the orthocenter seems identical to the one before it). So the critical points of the function $f(B,C)$ = oriented area of $ABC$ occur when $O$ is the orthocenter and are of the following types: 1 maximum of the oriented area when $O$ lies inside $ABC$ and $ABC$ is positively oriented 2 saddle points when $O$ lies outside $ABC$ 1 minimum when $O$ lies inside $ABC$ and $ABC$ is negatively oriented. This 1-2-1 pattern is good for cohomological reasons, because on $S^1$ (fixed $A,B$ and variable $C$) we expect 1 max and 1 min ($(1-t)$, and on $S^1 \times S^1$ we expect the distribution of critical point indices like $(1-t)^2$ = 1 min, 2 saddle, 1 max. The saddle points and the max/min are mirror images with respect to reflection in $OA$. So the consistency of mathematics is upheld, and only the nontrivial part of the problem remains. I should ask whether this problem in general involves solving an irreducible cubic or quartic.

fleeting_guest wrote: I should ask whether this problem in general involves solving an irreducible cubic or quartic. See my first post for that- the equation is the trig identity $\cos^2A+\cos^2B+\cos^2C+2\cos A\cos B\cos C=1$ for $A,B,C$ the angles of a triangle.

jhaussmann5 wrote: yetti wrote: $|\triangle ABC| \le \frac{R_A}{R} \cdot |\triangle OBC| + \frac{R_B}{R} \cdot |\triangle OCA| + \frac{R_C}{R} \cdot |\triangle OAB|$ The area $|\triangle ABC|$ is maximum at equality, i.e., when $O$ is the orthocenter. I don't quite see this. It's true that there is equality when $O$ is the orthocentre, but the right-hand side is not constant, as far as I can see. So, you can't conclude that the area is maximal simply when there is equality. The argument I put forward is either incorrect or insufficient. I do not see whether or how it can be fixed at the moment. Thanks to jmerry for and fleeting_guest for a correct argument. Yetti