$\mathbf{a)}$ There is such set. Look at set $M=\{a,b\}$. Before defining $a,b$ look at this polynomial. $$P(x)=2x^3+9x-15\sqrt{3}$$Since $P(0)<0<P(100)$, this polynomial has positive real root and let this root be $s$. Now choose $a=s^4$ and $b=3s^2$. So you can see that $5\sqrt{|a|}-\frac{2b}{3}=b=5\sqrt{|b|}-\frac{2a}{3}$. So set $M$ holds the given statement. $\mathbf{b)}$ I will show that there is no such set. Assume contrary. $M=\{t_1,t_2,...,t_k\}$ be set which holds statement, where $k\geq 3$ and $0<t_1<t_2<\cdots<t_k$. We have that all of $$5\sqrt{t_k}-\frac{2t_1}{3},5\sqrt{t_k}-\frac{2t_2}{3},...,5\sqrt{t_k}-\frac{2t_{k-1}}{3}$$are in $M$. Assume none of them equal to $t_k$. Since there is $k-1$ numbers, these numbers equal to $t_1,t_2,...,t_{k-1}$. So $5\sqrt{t_k}-\frac{2t_i}{3}=t_{k-i}$ for all $i=1,2,...,k-1$. So we have that $\\5\sqrt{t_k}-\frac{2t_1}{3}=t_{k-1}\ \text {and} \ 5\sqrt{t_k}-\frac{2t_2}{3}=t_{k-2} \implies t_{k-1}-t_{k-2}=\frac{2}{3}(t_2-t_1)$. $\\5\sqrt{t_k}-\frac{2t_{k-2}}{3}=t_{2}\ \text {and} \ 5\sqrt{t_k}-\frac{2t_{k-1}}{3}=t_{1} \implies t_{2}-t_{1}=\frac{2}{3}(t_{k-1}-t_{k-2})=\frac{4}{9}(t_2-t_1)$, which is contradiction. So we get that $1$ of the numbers $5\sqrt{t_k}-\frac{2t_1}{3},5\sqrt{t_k}-\frac{2t_2}{3},...,5\sqrt{t_k}-\frac{2t_{k-1}}{3}$ is equal to $t_k$ and since $t_k$ is the maximum between $t_i$s, we get $5\sqrt{t_k}-\frac{2t_1}{3}=t_k$. Now do same thing for $t_1$. Using minimality of $t_1$, we get $5\sqrt{t_1}-\frac{2t_k}{3}=t_1$. So we have $2$ nice equations to find contradiction. $$5\sqrt{t_k}-\frac{2t_1}{3}=t_k \ \text{and} \ 5\sqrt{t_1}-\frac{2t_k}{3}=t_1$$From sum and differnce of these equations we get $t_1+t_k=3(\sqrt{t_1}+\sqrt{t_k})=3\cdot 15=45 \implies 180=2\sqrt{t_1}\sqrt{t_k}\le t_1+t_k=45$, which is contradiction. So there is no such set.