By using complex numbers it is easily obtained that $\cos^2\angle GOM=\frac{\left(\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{8\pi\over 7}\right)^2}{3+2\left(\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{6\pi\over 7}\right)}=\frac{\left(-{1\over 2}\right)^2}{3+2\left(-{1\over 2}\right)}={1\over 8}$, but a synthetic solution would be nice.

Take on Argand Plane $G=1, A=\cos{2\pi\over 7}+i\sin{2\pi\over 7},B=\cos{4\pi\over 7}+i\sin{4\pi\over 7},D=\cos{8\pi\over 7}+i\sin{8\pi\over 7}$ Then $3M=A+B+D$ and $\cos^2\angle GOM=\Re \left(\frac{M}{|M|}\right)^2=\frac{\left(\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{8\pi\over 7}\right)^2}{\left(\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{8\pi\over 7}\right)^2+\left(\sin{2\pi\over 7}+\sin{4\pi\over 7}+\sin{8\pi\over 7}\right)^2}$ $\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{8\pi\over 7}=-1/2$ and $\sin{2\pi\over 7}+\sin{4\pi\over 7}+\sin{8\pi\over 7}=\frac{\sqrt{7}}{2}$ so my answer is also $\cos^2\angle GOM=\frac{1}{8}$ but I would like to know how Farenhajt got his denominator? Here I attach proofs of sums: $\sin{2\pi\over 7}+\sin{4\pi\over 7}+\sin{8\pi\over 7}=4\sin{2\pi\over 7}\sin{4\pi\over 7}\sin{6\pi\over 7}=4\sqrt{\sin(\frac{\pi}{7}) \cdot \sin(\frac{2\pi}{7}) \cdot ... \cdot \sin(\frac{6\pi}{7})}=4\sqrt{2^{1-7}\cdot 7}$ Las equality from here https://artofproblemsolving.com/community/c4h1209791

I see it now: $3$ comes from three expressions of type $\cos^2+\sin^2$ And this in bracket is $\cos{2\pi\over 7}\cos{4\pi\over 7}+\cos{4\pi\over 7}\cos{8\pi\over 7}+\cos{2\pi\over 7}\cos{8\pi\over 7}+\sin{4\pi\over 7}\sin{8\pi\over 7}+\sin{2\pi\over 7}\sin{8\pi\over 7}+\sin{2\pi\over 7}\sin{4\pi\over 7}=\cos{2\pi\over 7}+\cos{4\pi\over 7}+\cos{6\pi\over 7}$ Because of identity $\cos (a-b)=\cos a\cos b+\sin a\sin b$

Complex numbers

Complex is a skill issue. Let the circumradius be $1$, let $O'$ be the reflection of $O$ across $AB$, and let point $P$ be such that $ODPO'$ is a parallelogram. We see that $P$ is the orthocenter of $\triangle ABD$, so it suffices to compute $\cos^2 \angle GOP$. We do this in quite possibly the worst way possible. Let $PB$ and $O'A$ meet at point $Q$ and let $PQ$ and $OO'$ meet at $X$. We compute $\angle PXO' = 180^\circ - \angle O'PB - \angle XO'P = \pi - \frac{2\pi}{7} - \frac{3\pi}{7} = \frac{2\pi}{7}.$ Compute $\angle PO'A = \frac{3\pi}{7}$, hence we get $\angle O'AP = \frac{2\pi}{7}$. We compute $\angle O'AO = \frac{5\pi}{7}$, hence $\angle OAQ = \frac{2\pi}{7}$. As such both $OXQA$ and $PQO'A$ are cyclic, and so $A$ is the center of the spiral similarity sending $O'P$ to $OQ$. Now we notice that $AO' = AO$, so in fact $OQ = O'P$. Now compute \[ \angle OXA = \angle OXQ + \angle QXA = \angle OAQ + 180^\circ - \angle PXA = \angle OAQ + \angle PO'A = \frac{3\pi}{7} + \frac{2\pi}{7} = \frac{5\pi}{7}. \]Since $\angle XOA = \angle XQA = \frac{\pi}{7}$ it follows that $\frac{\pi}{7} = \angle XAO = \angle XQO$. Hence, the reflection of $O$ across $XQ$, say $Y$, lies on $QO'$. Now for the fun part. We recall the classic AIME config which states that $O'P = O'B = BY = YQ$. Now we see that $PO'$ is tangent to $(O'BQ)$ easily; hence, $PO'^2 = PB \cdot PQ$. But $PB \cdot PQ$ is the power of $P$ with respect to the circle centered at $Y$ with radius $QY$, so $PO'^2 = PY^2 - YQ^2$. Since $YQ = O'P = 1$ it follows that $PY^2 = 2$, and hence $PY = PO = \sqrt{2}$. Almost done now. We compute $\cos (180^\circ - \angle GOP)$. Vector summing the reflection of the orthocenter over $OG$ gives that the distance from $O$ to $P$ along the $OG$ component is $\frac{1}{2}$, so the cosine of this angle is $\frac{\frac{1}{2}}{\sqrt{2}}$. Thus, $\cos^2(\angle GOM) = \cos^2(\angle GOP) = \cos^2(180^\circ - \angle GOP) = \frac{1}{8}$.