AoPS - Problem

The answer is $\boxed{5}$. We obtain this value by noticing that there are at least $\binom{10}{2} = 45$ pairs of students who have the same book. Let there be $n$ total kinds of books, let $C_i$ denote the amount of times the $i$th book was bought, and let $X$ be the amount of pairs of students who have bought the same kind of book. We clearly have \[\sum_{i=1}^n \binom{C_i}{2} = X \geq 45 \Longrightarrow \sum_{i=1}^n \binom{C_i}{2} \geq 45\]Because there are $10$ students and each student buys $3$ books, we have that $\sum_{i=1}^n C_i = 10 \times 3 = 30$. From here it is not too difficult to see that if $C_i \leq 4$ for all $i$ then $\sum_{i=1}^n \binom{C_i}{2} \geq 45$ is impossible. Hence we obtain that $C_i \geq 5$ for some $i$ and then some construction efforts gives us that a maximum value of $5$ actually does work.