(b). Let \[X_k=\big\{\{s,t,u\}\mid\{s,t,u\}\subseteq\{1,2\dots,n\},s+t+u\equiv k\!\!\!\pmod n\big\}.\]Then $X_1$, $X_2 \dots$, $X_n$ form a partition of $\big\{A\mid A\subseteq\{1,2 \dots, n\}\big\}$. Thus, $\sum_{k=1}^n|X_k|=\text C_n^3$, and so there exists $k$ such that $|X_k|\ge\tfrac{\text C_n^3}n=\tfrac{(n-1)(n-2)}6$. Taking all elements of $X_k$, we reach the required quantity. Moreover, for two distinct sets $\{a,b,c\}$, $\{d,e,f\}$ that we took, assume $a=d$ and $b=e$, since $a+b+c\equiv d+e+f\pmod n$, so $c\equiv f\pmod n$, so $c=f$, so $\{a,b,c\}=\{d,e,f\}$, contradiction. We're done.

For part (a), let $T_{i,j}=1$ if $j\in A_i$ or $0$ if $j\notin A_i$. Note that when $j\ne k$ then $\sum_i T_{i,j} T_{i,k}\le 1$, since otherwise there exists $a\ne b$ such that $T_{a,j}=T_{a,k}$ and $T_{b,j}=T_{b,k}$, i.e. $A_a\cap A_b\supseteq\{j,k\}$ -- contradiction. Summing over pairs of columns: \[ \sum_{1\le j<k\le n} \left( \sum_{1\le i\le m} T_{i,j}T_{i,k}\right) \le \tfrac{n}2(n-1). \] Notice also that for each $i$, we have $\sum_{1\le j<k\le n} T_{i,j}T_{i,k} = 3$ because each row $T_{i,\star}$ contains exactly three $1$s (the rest are $0$s). Summing over rows: \[ \sum_{1\le i\le m} \left( \sum_{1\le j<k\le n} T_{i,j}T_{i,k} \right) = 3m. \] But the two double-sums are identical, so $3m\le \tfrac{n}2(n-1)$ as desired$\qquad\blacksquare$

Easier way for (a). Count the number of triples $(k_1,k_2,i)$ for which (a) $1\le k_1<k_2\le n$ and (b) $1\le i\le m$ with $k_1,k_2\in A_i$. Note that since $|A_i|=3$, we obtain that there are $3m$ such triples. On the other hand, each pair is contained in at most one such set (otherwise if there is $k_1,k_2$ with $k_1,k_2\in A_i\cap A_j$ for $i<j$ then $|A_i\cap A_j|\ge 2$), so the number of such triples is at most $\textstyle\binom{n}{2}$. Hence, $3m\le \textstyle \binom{n}{2}$, giving $m\le n(n-1)/6$.

grupyorum wrote: Easier way for (a). No, exactly the same way.

oolite wrote: grupyorum wrote: Easier way for (a). No, exactly the same way. For this problem, you don't really need incidence matrices and matrix multiplications; they complicate things as you see. There are problems though where I'd agree with you that using incidence matrices would be of tremendous benefit. Anyways.