Clearly $p\equiv 1\pmod{10}$, as $(5^{n})^5\equiv -1\pmod p$ and is not $-1$. If $p\equiv 3\pmod 4$, note that any primitive 10th root of unity, one of which is $5^n$, is not a quadratic residue (as if $g$ is a generator, and $g^k=5^n$, then $10k=p-1$ so $k$ is odd). However, by quadratic reciprocity, as $1$ is a quadratic residue mod $5$ and $5\equiv 1\pmod 4$, we are done.

Obviously p can be form of 4k+3 or 4k+1 and for the sake of contradiction let's assume that p=4p+3 simply we will get contradiction thus p= 1 (mod) 4

Is there a solution not involving Quadratic Reciprocity?

Let $n$ be a positive integer. Show that if p is prime dividing $5^{4n}-5^{3n}+5^{2n}-5^{n}+1$, then $p\equiv 1 \;(\bmod\; 4)$. Note that $p>2$ and \[p\mid\frac{5^{5n}+1}{5^n+1}.\]We split into cases. Case 1. $p\mid 5^n+1$. Then $\nu_p$ of the RHS thing is exactly $\nu_p(5)$ by LTE, so we must have $p=5$, which works. Case 2. $p\nmid 5^n+1$. Then the order of $5$ mod $p$ divides $10n$ but not $5n$. Thus this order uses every power of $2$ in $10n$. But note that $p\nmid 5^n-1$ since $p\nmid 5^{5n}-1$, so $p\nmid 5^{2n}-1$. Thus the order of $5$ mod $p$ also uses every power of $5$ in $10n$. Thus $p\equiv 1\pmod{10}$. Thus by Quadratic Reciprocity, $5$ is a QR mod $p$, so something has order divisible by $20$ mod $p$, so $20\mid p-1$. $\blacksquare$