By Fermat's Little Theorem, $k^p \equiv k \pmod{p}$, $k^{p-1} \equiv 1 \pmod{p}$. Set $k^{p-1}=bp+1$, $k^p \equiv bkp+k$. Then \begin{eqnarray*} \sum_{k=1}^{p-1} k^{2p-1} &=& k(bp+1)^2 \\ &\equiv& 2bpk + k \pmod{p^2}\\ &\equiv& 2k \cdot \left(k^{p-1}-1\right)+k \pmod{p^2}\\ &\equiv& 2k^p-k \pmod{p^2}\end{eqnarray*} Notice that $a^p + (p-a)^p \equiv 0 \pmod{p^2}$ from binomial theorem expansion, so we are left with $\sum_{k=1}^{p-1} -k \equiv -\left[\frac{p(p-1)}{2}\right] \pmod{p^2} \equiv \frac{p(p+1)}{2} \pmod{p^2}$.

Note that $2p-1$ is odd, by binomial expansion: $(p-i)^{2p-1}\equiv -i^{2p-1}-p*i^{2p-2}$ For $i=1$ to $\frac {p-1}{2}$ we replace $(p-i)^{2p-1}$ with the equivalence above to get LHS$\equiv -p(1+2^{2(p-1)}+...+\frac {p-1}{2}^{2(p-1)})$. By Fermat's last theorem $n^{p-1}\equiv 1$, hence $-(1+2^{2(p-1)}+...+\frac {p-1}{2}^{2(p-1)})\equiv -\frac {p-1}{2}\equiv \frac {p+1}{2} \pmod p$ and when we multiply everything straight across by $p$ we get: LHS$\equiv -p(1+2^{2(p-1)}+...+\frac {p-1}{2}^{2(p-1)})\equiv p\frac {p+1}{2} \pmod {p^2}$. Done