First note that if any of the $x_{i}$'s are $0$,so are all the other $x_{i}$'s.Thus there is one trivial solution for all n where $x_{i}=0$ for $i=0,1,2...n$. Next ,suppose no $x_{i}$ is $0$. Case $1$: $|a|>1$ and $n\geq 3$. Then as $P(X,Y)=P(Y,X)$ we have $P(x_{i},x_{i-1})=P(x_{i},x_{i+1})=0$ and so: $x_{i-1}$ and $x_{i+1}$ are the roots of the quadratic expression in $t,P(t,x_{i})=0$ $-(1)$. Hence,$x_{i-1}.x_{i+1}=(x_{i})^{2}$ .This implies that $x_{i}/x_{i+1}=k, k$ a real constant for all $i=0,1,...n$.By taking the product of all $x_{i}/x_{i+1}$ we get $k^{n}=1$,implying $k=1$ or $k=1,-1$ when $n$ is odd or even respectively.Also by $(1)$, $x_{i-1}+x_{i+1}+2ax_{i}=0$ but as $|a|>1$ and none of the $x_{i}$'s are $0$, $k$ cannot be $1$ or $-1$.Hence there is no non trivial solution. Case $2$: $a=-1$ The equation becomes $P(X,Y)=(X-Y)^{2}=0$.There is only one non trivial solution $x_{i}=p, p$ a non zero real constant.Also ,any other non trivial solution is equivalent to the above. Case $3$: $a=1$ The equation is now $P(X,Y)=(X+Y)^{2}=0$.The only non trivial solution is for an even $n$,that is $x_{2i+1}=p,x_{2i}=-p, p$ a non zero real constant.Any other non trivial aolution is equivalent to the above. Cheers Bhargav Ps:I haven't solved $n=2$ here but that is quite simple.

Observe that $ x_1^2 + 2ax_1x_2 + x_2^2 = 0 \Rightarrow x_2 = k x_1$, where $ k \in \{ - a + \sqrt {a^2 - 1}, - a - \sqrt {a^2 - 1} \}$ Therefore, $ x_2 = k_1 x_1$ $ x_3 = k_2 x_2$ $ \cdots$ $ x_1 = k_n x_n$ where each $ k_i \in \{ - a + \sqrt {a^2 - 1}, - a - \sqrt {a^2 - 1} \}$ Therefore, $ k_1 k_2 \cdots k_n = 1$ $ \Rightarrow ( - a + \sqrt {a^2 - 1})^r ( - a - \sqrt {a^2 - 1} )^{n - r} = 1$, where $ r$ is the number of $ k_i$'s equal to $ - a + \sqrt {a^2 - 1}$ $ \Rightarrow ( - a + \sqrt {a^2 - 1})^ {2r - n} = 1$ $ \Rightarrow 2r - n = 0$, since $ - a + \sqrt {a^2 - 1}$ is real and not equal to $ \pm 1$ $ \Rightarrow r = n/2$ Therefore number of distinct solutions is exactly the number ways of choosing $ n/2$ of the $ k$'s to be $ - a + \sqrt {a^2 - 1}$ and the rest $ - a - \sqrt {a^2 - 1}$ $ = \begin{cases} \binom{n}{n/2} +1 , \text{when n is even} \\ 1, \text{when n is odd} \end{cases}$ The extra 1 is to count the $ (0, 0, \cdots, 0)$ solution.