Cool problem. I will first establish $|M|=2$. To that end, let $M=\{x_1,\dots,x_n\}$ with $x_1<\cdots<x_n$, and $n\geqslant 3$. I claim that $x_1^3-\frac49x_n=x_1$. Assume the contrary that $x_1^3-\frac49 x_n>x_1$. Note that by the property of $M$, we have $$ \left\{x_1^3-\frac49 x_n,x_1^3-\frac49 x_{n-1},\dots,x_1^3-\frac49 x_2\right\}\subset M. $$Since $$ x_1<x_1^3-\frac49 x_n<x_1^3-\frac49 x_{n-1}<\cdots<x_1^3-\frac49 x_2\leqslant x_n, $$it holds that $x_1^3-\frac49 x_n = x_2$, $x_1^3-\frac49 x_{n-1}=x_3,\dots,x_1^3-\frac49 x_2 =x_n$. In particular, we obtain $x_3-x_2=\frac49 (x_n-x_{n-1})$, and $x_n-x_{n-1}=\frac49(x_3-x_2)$. But these two claims are certainly contradictory, unless $x_n=x_{n-1}$, which by assumption is not the case. Hence, $x_1^3-\frac49 x_n = x_1$. Now, a similar argument reveals also that $x_n^3-\frac49 x_1 =x_n$. Consider now the numbers $$ x_2^3 - \frac49x_n <x_3^3 -\frac49 x_n <\cdots < x_{n-1}^3 -\frac49 x_n. $$Note that, these numbers are all lower bounded by $x_1^3-\frac49 x_n$, and $x_n^3-\frac 49 x_1$. Thus, they must coincide with $x_2,\dots,x_{n-1}$. In particular, we conclude $x_2^3 - \frac49 x_n = x_2$. Consider, now, the numbers $$ x_2^3 - \frac49 x_{n-1}<x_3^3 - \frac49 x_{n-1}<\cdots x_{n-2}^3 - \frac49 x_{n-1}<x_n^3 -\frac49 x_{n-1}. $$Note that these numbers are all bounded below by $x_1^3-\frac49 x_n=x_1$ and bounded above by $x_n^3-\frac49 x_1 = x_n$. Hence, they do coincide with $x_2,\dots,x_{n-1}$. In particular, $x_2^3-\frac49 x_{n-1}=x_2$. But we have found above that $x_2^3 -\frac49 x_n =x_2$ as well. These two statements clearly yield a contradiction. Thus, $|M|\leqslant 2$. Now, let $M=\{a,b\}$ with $a<b$. By construction $a^3-\frac49 b<b^3-\frac49 a$, and both belong to the set. We thus obtain $a^3-\frac49 b= a$ and $b^3-\frac49 a=b$. In particular, we get $a^2+ab+b^2 =\frac59$, and $a^3+b^3=\frac{13}{9}(a+b)$. Now if $a+b=0$, then we obtain the set $$ \left\{-\frac{\sqrt{5}}{3},\frac{\sqrt{5}}{3}\right\}. $$Let now $a+b\neq 0$, thus $a^2-ab+b^2 = \frac{13}{9}$. This, together with $a^2-ab+b^2 = \frac59$ then yields $ab=-4/9$, and thus $$ a+b = \pm 1\quad\text{and}\quad a-b\in \pm \frac{\sqrt{17}}{3}. $$From here, we deduce the following two more sets: $$ \left\{\frac{1-\sqrt{17}}{6},\frac{1+\sqrt{17}}{6}\right\},\quad\text{and}\quad \left\{\frac{-1-\sqrt{17}}{6},\frac{-1+\sqrt{17}}{6}\right\}. $$These are the only solutions.