If one number is positive, take biggest $x_i$, then $x_i^2+2ax_i+b^2$ is maximal again, hence solutions are $(x_i,x_i,\cdots,x_i)$ where $x_i$ is positive solution of $x_i^2+(2a-1)x_i+b^2$ if possible. Otherwise, they are all negative, but with AM-GM the LHS is positive, so no solution more.

SCP wrote: If one number is positive, ... Otherwise, they are all negative, .... I think you have forgotten that the variables may be complex, something that I would guess that the OP has forgotten to specify. However, since we don't know that, complex solutions are still missing.

I checked the question again; there is no specification as to whether $x_i$ are real or not.

Defining $f(x) = x^2 + 2ax + b^2$, the system becomes $x_2 = f(x_1)$, $x_3 = f(x_2)$, $\ldots$, $x_n = f(x_{n-1})$, $x_1 = f(x_n)$, hence $x_1 = f^n(x_1)$ (where $f^n$ is the $n$-th iterate of $f$). Thus, the complex solutions are generated by those of the equation $f^n(z) - z = 0$, a polynomial equation of degree $2^n$. By the fundamental theorem of algebra, there are full $2^n$ solutions (with possible multiplicities). However, the specific conditions on the real numbers $a$ and $b$, plus my (and others') experience with such systems being presented with just real roots in mind, point to considering just real solutions. Then the typical study of such iterated systems revolves around the real roots of the equation $x^2 + (2a-1)x + b^2 = 0$, which can be none, one (when the graph of $f$ is tangent to the first bisectrix), or two. The discriminant is $\Delta = (2a-1)^2 - 4b^2$ and needs be non-negative to have any real root(s). This is impossible if $2a - 1 \geq 0$, so we need $2a-1<0$ and then $2(a+b) \leq 1$ (so in fact $4a \leq 1$). The allure of the graph points to the only orbits being those of length $1$, thus having all variables $x_k$ being equal (to one of the above mentioned roots, when $a$ and $b$ correspond).

bzprules wrote: I checked the question again; there is no specification as to whether $x_i$ are real or not. According to a Turkish Math Journal, the problem statement is only "given real numbers $b \geq a$". According to the book Mathematical Contests 1995-1996 Olympiad Problems and Solutions from around the World, the problem statement is "given real numbers $b \geq a > 0$" with a note stating that "Corrected"

Strangely enough, this is sitting with no solutions. We find all real $(x_1,\dots,x_n)$ solutions to the system. First, I claim $\textstyle\min_i x_i\ge 0$. Indeed, note that \[ x_i = x_{i-1}^2 + 2ax_{i-1}+b^2=(x_{i-1}+a)^2 + (b^2-a^2)\ge 0. \]Next, we study $x_1$ and $x_2$. If $x_1=x_2$ then $x_1^2+2ax_1+b^2 = x_2^2+2ax_2+b^2\implies x_2=x_3$. Iterating, we get $x_i$ are equal. Now assume $x_1>x_2$. Then, $x_2>x_3>\cdots>x_n$, yielding $x_1>x_n$. But then as $x_1>x_2$, we get $x_n>x_1$, a contradiction. Likewise, $x_2>x_1$ is impossible. Hence $x_i$ are all equal. Under this, we arrive at \[ x^2+(2a-1)x + b^2 = 0 \implies x=\frac{1-2a \pm \sqrt{(1-2a)^2-4b^2}}{2}. \]Note that $(1-2a)^2-4b^2 = (1-2a-2b)(1-2a+2b)$ and since $1-2a+2b>0$, it must be the case $1-2a-2b\ge 0\implies a+b\le 1/2$. Once $a+b\le 1/2$, we have $2a\le 1/2\implies 1-2a\ge 1/2$ so both solutions above work.