We first rotate point $B,C,P$ $60^o$ anticlockwise from point $A$ so that $B\to C$ , $C\to C'$ , $P\to P'$ . Since $AP=AP'=5$ and $\angle PAP'=60^o$ , so $\triangle PAP'$ is equilateral . Hence $PP'=5$ In triangle $PCP'$ , using cosine rules to get $\angle P'PC=60^o$ , hence the angle $\angle APC=120^o$ So $AC^2=AP^2+AC^2-2AP\cdot AC\cos 120^o=25+64-2(5)(8)\frac{1}{2}=129$ $AC=\sqrt{129}$

See also http://www.mathlinks.ro/Forum/viewtopic.php?t=77511 http://www.mathlinks.ro/Forum/viewtopic.php?t=67254 Darij