I think the answer is 25. The condition $11|(ax+by)(ay+bx)$ is equivalent to $\{ \frac{a}{b},\frac{b}{a} \}$ and $\{ -\frac{x}{y},-\frac{y}{x} \}$ have no common element under $\pmod {11}$ .We know that if $a,b \in M,a \neq b$,then the set $\{ \frac{a}{b},\frac{b}{a} \}$ under $\pmod {11}$ can be $\{2,6\},\{3,4\},\{5,9\},\{7,8\},\{10,10\}$,while the last one corresponding 5 $\{a,b\}$s and the other corresponding to $10$.We also know that $\{2,6\}$ and $\{5,9\}$,$\{3,4\}$ and $\{7,8\}$ can't appear at the same time,so the maximum value should be$10+10+5=25$.