Suppose a graph $G$ with $n$ vertices and $|E|$ edges has domination number $\gamma$ Then $$|E|\leq \frac{1}{2}(n-\gamma)(n-\gamma+2)$$This is known as Vizing's theorem. In our case the domination number is at least $3$. Hence $$|E|\leq \left\lfloor \frac{(n-3)(n-1)}{2}\right\rfloor $$The equality is attained when $G$ is graph with one isolated vertex $v_0$. All the other vertices $v_1,v_2,\dots,v_{n-1}$ are connected except $v_1v_2, v_3v_4,\dots $ where in case $n$ - odd, the last pair is $v_{n-2}v_{n-1}$, and for $n$ -even, the last pair is $v_{n}v_1$. One may see something similar here.

We examine the graph $G$ which is supposed to be the complement of the graph specified. It is known that for every two vertex in $G$ ,there exists a vertex that is connected with both of em. Let $E$ denote the number of edges in $G$. It can also be observed that $\delta (G) \ge 1$. We count the number of triples $(A,B,C)$ where these three are vertices in $G$ and $C$ is connected with both $A,B$. Fixing $C$ we see that this is $\sum \binom{d_{i}}{2}$ Also this has to be $\ge \binom{n}{2}$. We see that whenever we have two integers $a,b$ both at least $1$, $\binom {a}{2} + \binom {b}{2} \le \binom{a+b-2}{2} +1$. Using this by combining the vertices in descending order of their degree , $\sum \binom{d_{i}}{2} \le n-1 +\binom{2E - 2(n-1)}{2}$. Now, using the above obtained inequality we get that , $\binom{n-1}{2} \le \binom{2E - 2(n-1)}{2}$, which gives us that $E \ge 3 \frac{n-1}{2}$. Now examining the given graph, we see that the number of edges $\le n \frac{n-1}{2} -3 \frac{n-1}{2}$, which gives us the desired upper bound of $(n-1) \frac{(n-3)}{2}$.