Let be $A,B,C$ three consecutive vertices of the regular hexagon, $AB=BC=a; M$ a point inside the hexagon such that $MA=MB=1,MC=2; MN\perp AB, N\in AB; \angle{MAB}=\angle{MBA}=\alpha\in(0^\circ,90^\circ)$.
Then: $AN=BN=\dfrac{a}{2}; \angle{ABC}=120^\circ;\angle{MBC}=120^\circ-\alpha$.
In the triangle $MAN: AN=AM\cos\alpha\Longrightarrow a=2\cos\alpha$.
Using the cosine theorem in the triangle $MBC$:
$MB^2+BC^2-2MB\cdot BC\cos{(\angle{MBC})}=MC^2\Longrightarrow$
$\Longrightarrow 1+4\cos^2\alpha-4\cos\alpha\cdot\cos{(120^\circ-\alpha)}=4\Longrightarrow$
$\Longrightarrow 4\cos^2\alpha-2[\cos{(120^\circ-2\alpha)}+\cos120^\circ]=3\Longrightarrow$
$\Longrightarrow 2+2\cos2\alpha-2\left(-\dfrac{1}{2}\cos2\alpha+\dfrac{\sqrt3}{2}\sin2\alpha-\dfrac{1}{2}\right)=3\Longrightarrow$
$\Longrightarrow 3\cos2\alpha-\sqrt3\sin2\alpha=0\Longrightarrow \tan2\alpha=\sqrt3\Longrightarrow \alpha=30^\circ$.
Results the length of the hexagon's side:
$a=2\cos\alpha=\sqrt3$.