I don't think this problem is stated correctly. If the relation only holds for all $n \geq 1$, there is no information at all about $x_0$. On the other hand, if it holds also for $n=0$ then $1=x_1(x_0-|x_0|)=0$ since $x_0 > 0$.

Edited; look again.

Do you mean "of a quadratic equation with integer coefficients"? Or perhaps "with rational coefficients"?

Let $f(x) = \frac{1}{x-\lfloor x \rfloor}$. Then for each positive $x$, the continued fraction of $f(x)$ is just the continued fraction of $x$ shift by $1$. Therefore, if $(x_n)$ defined by $x_{n+1} = f(x_n)$ is periodic, then the continued fraction of $x_0$ is periodic meaning that $x_0$ is a root of a integer coefficient quadratic polynomial which is irreducible over $\mathbb{Z}[x]$. Converse is almost true if we replace `periodic' by `eventually periodic'. To be precise, if $x_0$ is a zero of an irreducible polynomial in $\mathbb{Z}[x]$, then it has the continued fraction which is periodic from at some point. Therefore $(x_n)$ is periodic eventually. For reference, cite https://en.wikipedia.org/wiki/Periodic_continued_fraction. For other appearance of the theorem applied questions, cite https://artofproblemsolving.com/community/c6h1543871p9356329 So at my opinion, this kind of questions are not suitable for math competitions.

seoneo wrote: not suitable for math competitions. In some Romanian selective exercises it's the reasoning mannerisms that the problem makers are looking for, not necessarily the complete solution. And, if the student solves an open problem... good for him.

The harder part (the converse) studied here for simple square roots. It's not hard to generalize the proof for non-rational roots of generic quadratic equations with integer coefficients.