Maybe something like vieta jumping helps to this?

sqing wrote: Find the smallest positive number $\lambda $ , such that for any complex numbers ${z_1},{z_2},{z_3}\in\{z\in C\big| |z|<1\}$ ,if $z_1+z_2+z_3=0$, then $$\left|z_1z_2 +z_2z_3+z_3z_1\right|^2+\left|z_1z_2z_3\right|^2 <\lambda .$$

We claim that $\boxed{\lambda = 1}$. First, we prove that $\lambda \ge 1$. Take $z_1=1-\epsilon$, $z_2=0$, and $z_3=\epsilon-1$, where $\epsilon$ is a small positive real. We would have $\text{L.H.S} = |z_1z_3|^2 = (1-\epsilon^2)^2$, which can be arbitrarily close to $1$. Good. Now we prove that $\lambda = 1$ will work. Take $z_k = r_k(\cos \theta_k + i \sin \theta_k)$ for $k=1,2,3$. From the given condition, we have $\sum r_i \cos \theta_i = \sum r_i \sin \theta_i = 0$. Now this gives us the two equalities - $$r_1 \cos \theta_1 + r_2 \cos \theta_2 = -r_3 \cos \theta_3$$$$r_1 \sin \theta_1 + r_2 \sin \theta_2 = -r_3 \sin \theta_3$$Squaring both then adding them up together with the difference-of-cosines formula, we have $$r^2_1+r^2_2 + 2r_1r_2 \cos (\theta_2 - \theta_1) = r^2_3$$which gives us $$\cos (\theta_2 - \theta_1) = \frac{r^2_3-r^2_1-r^2_2}{2r_1r_2}$$Now $$2r^2_1r^2_2 \cos (2\theta_2 - 2\theta_1) = 2r^2_1r^2_2 (2 \cos^2 (\theta_2-\theta_1) - 1) = (r^2_3-r^2_1-r^2_2)^2 - 2r^2_1r^2_2 = r^4_1+r^4_2+r^4_3 - 2r^2_1r^2_3 -2r^2_2r^2_3$$Adding cyclic permutations gives us $$\sum_{1\le i<j \le 3} 2r^2_ir^2_j \cos (2\theta_j - 2\theta_i) = 3(r^4_1+r^4_2+r^4_3) - 4(r^2_1r^2_2+r^2_2r^3_2+r^2_3r^2_1)$$Since $z_1+z_2+z_3=0$, we can swap $z_1z_2+z_2z_3+z_3z_1$ with $\frac{1}{2}(z^2_1+z^2_2+z^2_3)$. This gives us $$\text{L.H.S} = \frac{1}{4}|z^2_1+z^2_2+z^2_3|^2 + |z_1z_2z_3|^2 = \frac{1}{4}|\sum r^2_i \cos 2\theta_i + i\sum r^2_i \sin 2\theta_i|^2 + r^2_1r^2_2r^2_3 = \frac{1}{4}((\sum r^2_i \cos 2\theta_i)^2+(\sum r^2_i \sin 2\theta_i)^2) + r^2_1r^2_2r^2_3$$$$ = \frac{1}{4}(r^4_1+r^4_2+r^4_3 + 2\sum_{1 \le i < j\le 3} r^2_ir^2_j \cos (2\theta_j - 2\theta_i)) + r^2_1r^2_2r^2_3= \frac{1}{4}(r^4_1+r^4_2+r^4_3+3(r^4_1+r^4_2+r^4_3) - 4(r^2_1r^2_2+r^2_2r^3_2+r^2_3r^2_1)) + r^2_1r^2_2r^2_3$$$$= r^4_1+r^4_2+r^4_3 - (r^2_1r^2_2+r^2_2r^3_2+r^2_3r^2_1) + r^2_1r^2_2r^2_3 \le r^2_1+r^2_2+r^2_3 -(r^2_1r^2_2+r^2_2r^3_2+r^2_3r^2_1) + r^2_1r^2_2r^2_3 = 1-(1-r^2_1)(1-r^2_2)(1-r^2_3) \le 1$$which implies that $\lambda=1$ works. Done. $\blacksquare$

Another sufficiently distinct solution: Objective is to solve this problem as elementary as possible with minimal knowledge of complex numbers. Denote $r_1, r_2, r_3$ be the modulus of $z_1, z_2, z_3$. Observe for any two complex numbers $z_1, z_2$, $|z_1z_2|^2=|z_1|^2|z_2|^2$, because $|z_1z_2|^2=|(x_1+y_1i)(x_2+y_2i)|^2=(x_1x_2-y_1y_2)^2+(x_1y_2+x_2y_1)^2=(x_1^2+y_1^2)(x_2^2+y_2^2)=|z_1|^2||z_2|^2$. So the originally hard to expand term $|z_1z_2z_3|^2$ is just $r_1^2r_2^2r_3^2$. Now we count $|z_1z_2+z_2z_3+z_3z_1|^2$ in two ways. The aim is to prove $r_1^4+r_2^4+r_3^4\ge 2(r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2)$. Time to expand. Take the original expression, then $$z_1z_2+z_2z_3+z_3z_1=\sum_{cyc}{(x_1+y_1i)(x_2+y_2i)}=\sum_{cyc}{(x_1x_2-y_1y_2)}+i\sum_{cyc}{(x_1y_2+x_2y_1)}$$$$\Rightarrow |z_1z_2+z_2z_3+z_3z_1|^2=(\sum_{cyc}{(x_1x_2-y_1y_2)})^2+(\sum_{cyc}{(x_1y_2+x_2y_1)})^2$$Every term $-2(x_ix_{i+1})(y_iy_{i+1})$ from the first bracket, can be paired with $2(x_iy_{i+1})(x_{i+1}y_i)$ from the second bracket, so the sum is just equal to $$\sum_{cyc}(x_1^2+y_1^2)(x_2^2+y_2^2)=r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2$$ Now, use the condition that $z_1+z_2+z_3=0$, then $z_1^2+z_2^2+z_3^2=-2(z_1z_2+z_2z_3+z_3z_1)$. Consequently, $|z_1z_2+z_2z_3+z_3z_1|^2=\frac{1}{4}|z_1^2+z_2^2+z_3^2|^2$. We get $$z_1^2+z_2^2+z_3^2=\sum_{cyc}{(x_1+y_1i)^2}=\sum_{cyc}{(x_1^2-y_1^2)}+2i\sum_{cyc}{x_1y_1}$$$$\Rightarrow |z_1^2+z_2^2+z_3^2|^2=(\sum_{cyc}{(x_1^2-y_1^2)})^2+4(\sum_{cyc}{x_1y_1})^2$$Observe that it is equal to $$(\sum_{cyc}{(x_1^2+y_1^2)})^2-(2\sum_{cyc}{x_1^2})(2\sum_{cyc}{y_1^2})+4(\sum_{cyc}{x_1y_1})^2$$And by Cauchy Scwarz Inequality we get $4(\sum_{cyc}{x_1^2})(\sum_{cyc}{y_1^2})\ge 4(\sum_{cyc}{x_1y_1})^2$, so $$|z_1^2+z_2^2+z_3^2|^2\le (\sum_{cyc}{(x_1^2+y_1^2)})^2=(\sum_{cyc}{r_1^2})^2=(r_1^4+r_2^4+r_3^4)+2(r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2)$$ Thus $$r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2=|z_1z_2+z_2z_3+z_3z_1|^2=\frac{1}{4}|z_1^2+z_2^2+z_3^2|^2\le \frac{1}{4}[(r_1^4+r_2^4+r_3^4)+2(r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2)]$$$$\Rightarrow r_1^4+r_2^4+r_3^4\ge 2(r_1^2r_2^2+r_2^2r_3^2+r_3^2r_1^2)$$as claimed. With this result, we will prove that $\lambda=1$ is the best constant. To see why it is necessary, just take $z_1=-z_2=a<1, a\in \mathbb{R}, z_3=0$, then clearly $\lambda=a^2$ in this case, so we may take $a$ infinitesimally close to $1$, hence $\lambda\ge 1$ is necessary. To prove $\lambda =1$ is sufficient, just note that $r_k^4\le r_k^2$, so $$|z_1z_2+z_2z_3+z_3z_1|^2+|z_1z_2z_3|^2=\sum_{cyc}{r_1^2r_2^2}+r_1^2r_2^2r_3^2\le \sum_{cyc}{r_1^4}-\sum_{cyc}{r_1^2r_2^2}+r_1^2r_2^2r_3^2\le \sum_{cyc}{r_1^2}-\sum_{cyc}{r_1^2r_2^2}+r_1^2r_2^2r_3^2=1-(1-r_1^2)(1-r_2^2)(1-r_3^2)\le 1$$ And we are done

We claim the answer is $\lambda=1$.For the sake of brevity, we allow $|z_i|$ to be 1, correspondingly we allow the equality case to hold in the inequality we wish to prove. Now, to see $\lambda=1$ is the best bound, just pick $z_i={\omega}^i$, where $\omega=-\frac{1}{2}+\frac{\sqrt{3}i}{2}$. To prove the inequality, there is one trick to make this so much simpler, just spin $z_i$($1\leq i\leq 3$) with some angle, and possibly with some expansion, and we may assume that $z_1=1$. Well, the rest is fairly easy, I believe that you can work is out in 20 minutes!lol

sqing wrote: Find the smallest positive number $\lambda $ , such that for any complex numbers ${z_1},{z_2},{z_3}\in\{z\in C\big| |z|<1\}$ ,if $z_1+z_2+z_3=0$, then $$\left|z_1z_2 +z_2z_3+z_3z_1\right|^2+\left|z_1z_2z_3\right|^2 <\lambda .$$ Let $z_1\rightarrow 1$, $z_2\rightarrow \omega$, $z_4\rightarrow\omega ^2$, where $\omega =e^{\frac{2\pi i}{3}}$. Then we can get $\lambda\geqslant 1$. Define $|z_1|^2=r_1$, $|z_2|^2=r_2$, $|z_3|^2=r_3$. Then $z_1\overline{z_2}+z_2\overline{z_1}=(z_1+z_2)(\overline{z_1}+\overline{z_2})-|z_1|^2-|z_2|^2=|z_1+z_2|^2-|z_1|^2-|z_2|^2=r_3-r_1-r_2$. We can get $z_2\overline{z_3}+z_3\overline{z_2}=r_1-r_2-r_3$, $z_3\overline{z_1}+z_1\overline{z_3}=r_2-r_3-r_1$ using the same method. $\therefore\begin{aligned}\left|z_1z_2 +z_2z_3+z_3z_1\right|^2+\left|z_1z_2z_3\right|^2&=|z_1z_2|^2+|z_2z_3|^2+|z_3z_1|^2+|z_1|^2(z_2\overline{z_3}+z_3\overline{z_2})+|z_2|^2(z_3\overline{z_1}+z_1\overline{z_3})+|z_3|^2(z_1\overline{z_2}+z_2\overline{z_1})+r_1r_2r_3\\&=r_1r_2+r_2r_3+r_3r_1+r_1(r_1-r_2-r_3)+r_2(r_2-r_3-r_1)+r_3(r_3-r_1-r_2)+r_1r_2r_3\\&=r_1^2+r_2^2+r_3^2-r_1r_2-r_2r_3-r_3r_1+r_1r_2r_3\\&<r_1+r_2+r_3-r_1r_2-r_2r_3-r_3r_1+r_1r_2r_3\\&=(r_1-1)(r_2-1)(r_3-1)+1<1.\end{aligned}$