We can just use McCoy's theorem. Since X and Y commute, the eigenvalues of XY are products of pairs of eigenvalues of X and Y. Since M is the set of matrices with eigenvalues of modulus smaller than 1, any product of eigenvalues of X and Y from M has modulus smaller than 1, thus proving that XY is in M.

is it possible to do this without stating and using this theorem. Is there any other way to do this question. This theorem seems more theoretical that I am unable to catch it and also I am not finding any relevant resource to learn this theorem.

McCoy's theorem has nothing to do with this (it is a condition for simultaneous triagulation of two matrices $A$ and $B$ based on the eigenvalues of the matrices that are polynomial in $A$ and $B$.) Here, $X$ and $Y$ commute, and so are simultaneously triangulable, which easily proves the claim.