Using polar coordinates, let: $a = r \cdot cos(t), b = r \cdot sin(t)$ which transform our given system of equations into: $r^3[(cos(t))^3 - 3cos(t)*(sin(t))^2] = r^3[(cos(t))^3 - 3cos(t)(1-(cos(t))^2)] = r^3[4(cos(t))^3 - 3cos(t)] = r^3cos(3t) = 8$ (i); $r^3[(sin(t))^3 - 3sin(t)*(cos(t))^2] = r^3[(sin(t))^3 - 3sin(t)(1-(sin(t))^2)] = -r^3[-4(sin(t))^3 + 3sin(t)] = -r^3sin(3t) = 11$ (ii). Substitution of (i) into (ii) yields an equation in $t$, or: $\frac{sin(3t)}{cos(3t)} = -\frac{11}{8}$; or$ tan(3t) = -\frac{11}{8}$; or $3t = arctan(-\frac{11}{8})$ (iii). If we return to (i) we find that: $r^3cos(3t) = r^3cos(arctan(-\frac{1}{8}) = r^3cos(arccos(\frac{8}{\sqrt{185}}) = \frac{8r^3}{\sqrt{185}} = 8$; or $r = 185^{1/6}$ (iv) and $a^2 + b^2 = r^2 = 185^{1/3}$.

Brute force answer with SageMath : R.<a,b,c> = PolynomialRing(QQ,"a,b,c") I=R.ideal([a^3-3*a*b^2-8,b^3-3*b*a^2-11, c-a^2-b^2]) I.elimination_ideal([a,b]) Ideal (c^3 - 185) of Multivariate Polynomial Ring in a, b, c over Rational Field so c = 185^(1/3) More clever answer : (a + i*b) ^3 = 8 - 11*i, so |a+ib|^2 = (64 + 121) ^(1/3).