Suppose every $x_i$ is $0$ or $1$. Suppose there are $m$ $1s$. Then we have $m(4p+1)=4m^2+4p+1$, so $4m^2=(4p+1)(m-1)$. Now, since $4p+1$ is odd, $4\mid (m-1)\implies m$ is odd $\implies$ 8 does not divide $m-1$. Since $m$ and $m-1$ are relatively prime, no prime greater than $2$ can divide $m-1$. Hence $m=5$ and $p=6$ which is a possible solution. If $x_i$ is a solution, than $-x_i$ is also a solution. Assume that $\sum x_i\ge 0$. If some $x_i\neq 0, 1$, then either some $x_i$ is negative, or some $x_i>1$. In either case we have $x_i^2>x_i$, so $\sum x_i<\sum x_i^2$. Assigning $S=\sum x_i$ we have $S+1\le\sum x_i^2=\frac{4S^2}{4p+1}$, so $S\ge\frac{4p+1}{4}\implies S\ge p+1$. By Cauchy, we have $S^2\le p\left(\sum x_i^2\right)=4p/(4p+1)S^2+p$, so $S^2\le p(4p+1)\implies S\le 2p$. Therefore $1<S/p\le 2$. We have $\sum (x_i-S/p)^2=\left(\sum x_i^2\right)S^2/p=1-S^2/(p(4p+1))<1$. Hence each $|x_i-S/p|<1\implies 0<x_i<3$. But $x_i$ is an integer, so it is $1$ or $2$. As previously, suppose there are $m$ $1s$. Then there are $p-m$ $2s$ and we have $4mp-3m-4m^2-1=0$. Hence $m$ divides $1\implies m=1\implies p=2$. Hence the only values of $p$ are $6$ and $2$. We are done.