For part a), first note that given triangle $ABC$, let $D$ be a point on ray $BC$ such that $C$ is between $B$, and $D$ (or $C=D$). Then $AB^2>BC\cdot BD$ iff $\angle BAC<\angle ADB$, $AB^2<BC\cdot BD$ iff $\angle BAC>\angle ADB$, and $AB^2=BC\cdot BD$ iff $\angle BAC=\angle ADB$. Now let the points be $A=A_1, A_2,...,A_n=B$. Construct a point $C$ on the perpendicular bisector of $A_1A_n$ such that $\angle A_1CA_n=\frac{n-1}{n+1}\cdot 180^{\circ}$. Then $A_1CA_n$ is isoceles so $\angle CA_1A_n=\angle CA_nA_1=\frac{1}{n+1}\cdot 180^{\circ}$. Note for any point $A_i$, it is clear that the inversion relation must map $A_1$ to $A_{i-1}$ and $A_{i+1}$ to $A_n$. Look at $\angle A_1CA_2$. Note that if $\angle A_iCA_{i+1}<$(or $>$ or $=$)$\frac{1}{n+1}\cdot 180^{\circ}=\angle CA_1A_{i+1}$, then $A_{i+1}C^2>$(or $>$ or $=$)$A_{i+1}A_1\cdot A_iA_1=A_{i+1}A_n\cdot A_{i+2}A_n\Rightarrow \angle A_{i+1}CA_{i+2}<$(or $>$ or $=$)$\angle CA_nA_{i+2}=\frac{1}{n+1}\cdot 180^{\circ}$. Hence if $\angle A_1CA_2<\frac{1}{n+1}\cdot 180^{\circ}$, then $A_1CA_n<\frac{n-1}{n+1}\cdot 180^{\circ}$, and similarly if $\angle A_1CA_2>\frac{1}{n+1}\cdot 180^{\circ}$, then $\angle A_1CA_n>\frac{n-1}{n+1}\cdot 180^{\circ}$, both are contradiction. So we must have $\angle A_1CA_2=\frac{1}{n+1}\cdot 180^{\circ}$, which implies that $A_iCA_{i+1}=\frac{1}{n+1}\cdot 180^{\circ}$, which means set $A_i$ is unique. It remains to check that this will work. For any point $A_i$, invert w.r.t circle of radius $A_iC$. Then note that for $1\leq k\leq \frac{i}{2}$, $\angle A_{i-k}CA_k=\frac{k}{n+1}\cdot 180^{\circ}=\angle CA_kA_i$, so $A_iA_{i-k}\cdot A_iA_k=A_iC^2$, hence $A_k, A_{i-k}$ map to each other after inversion. Similarly, points $A_{k}, A_{i+n-k}$ map to each other after inversion, hence this configuration of $n$ points work.

Very nice problem!

Can someone clarify the case $n=3$ for part (a)?? According to me it is not possible: Let the points be $A,C,B$ in that order. Note that the only other point other than $C$ on ray $CB$ in $S$ is $B$. Thus, $r = CB$. Similarly $CA=r$. But then with center $A$, $B$ is not mapped to any admissible point. Please help

$B$ maps to $C$ with radius $\sqrt{2}r$

O! I see now, I had misunderstood the problem: sorry.

Anyone solve this one

(a) First we'll construct the desired set. Start with a regular $(n+1)-$gon $A_0 A_1 A_2 \cdots A_n.$ Invert about $A_0$ with radius $1$, let the image of $A_i$ be $B_i$ for $1 \le i \le n.$ Then perform a spiral similarity so that $B_1 = A, B_n = B.$ Now, let's show that this is unique. Let $S = \{A, P_1, P_2, \cdots, P_{n-2}, B\}$ be a very set where $A, P_1, P_2, \cdots, P_{n-2}, B$ are on line segment $AB$ in this order. Note that $P_1$ uniquely determines the rest of the $P_i$'s. Indeed for $ \ge 2$, if we know $P_{i-1}, P_i$, then $P_{i+1}$ is just the unique point on segment $P_iB$ so that $P_iP_{i+1} \cdot P_i B = P_i P_{i-1} \cdot P_i P_1.$ Hence, we can view the positions of $P_2, P_3, \cdots, P_{n-2}$ as a function of $P_1.$ It's easy to check that these functions all move towards $B$ as $P_1$ does, and it is therefore clear by monotonicity that there is a unique choice of $P_1$ which gives rise to a $very$ set. (b) Too hard

(a) Rename $A$ to $X_1$ and $B$ to $X_n$ and let $X_2,X_3,\dots,X_{n-1}$ be the other points in the set $S.$ Let the circle of inversion centered at $X_i$ be $\omega_i.$ It is easy to see $\omega_i$ must contain $X_{i+1}$ in its interior or boundary whenever $1\le i\le n-1.$ Thus $\omega_i,\omega_{i+1}$ either intersect twice or $\omega_i$ contains $\omega_{i+1}$ in its interior or boundary. But clearly $\omega_1$ cannot contain $\omega_n$ in its interior or on its boundary, thus there is some $k$ for which $\omega_k,\omega_{k+1}$ intersect at some point $A$ not collinear with the $X_i.$ Suppose $i<k.$ We have \[\angle X_kAX_i=\angle AX_{k-i}X_k=\angle AX_{k-i}X_{k+1}=\angle X_{k+1}AX_{i+1}\]by similar triangles, so $\angle X_iAX_{i+1}=\angle X_kAX_{k+1}.$ Similarly if $j>k$ we also may get $\angle X_jAX_{j+1}=\angle X_kAX_{k+1}.$ Additionally, we have $\angle AX_1X_{k+1}=\angle X_kAX_{k+1}=\angle AX_nX_k$ by similar triangles, so we have $n+1$ equal angles. In fact the equal angles show that all $\omega_i$ pass through $A.$ Upon inverting about $A$ we see $AX_1'X_2'\dots X_n'$ must be a regular $n+1$-gon, which is enough to show that such a very set must be unique. Additionally we can easily check that this works, so we are finished. (b) We claim the answer is $5$ which is achieved by a square together with its center, which clearly works. Consider a side of the convex hull of the set, and let its endpoints be $A,B.$ Suppose $C$ also lies on this side. Let $X$ be the point in the set closest to the side. By assumption, there are no other points along ray $XA,$ and similarly for $XB$ and $XC.$ Thus the circle centered at $X$ has to pass through $A,B,C$ which is impossible, thus such a point $C$ does not exist. Now consider two consecutive sides $AB,AC$ of the convex hull. Since lines $AB$ and $AC$ only contain two points each, the circle centered at $A$ must pass through both $B$ and $C,$ thus $AB=AC$ and thus we get that the convex hull is equilateral. As before let $AB,AC$ be sides of the convex hull. From part (a) together with checking the case for a segment with just two points we get that the inverses of points on line $BC$ about the circle centered at $A$ together with $A$ itself must form a regular polygon, with $B,A,C$ as three of its consecutive vertices. The fact that this regular polygon must exist implies that $\angle BAC=180-\frac{360}n$ for some positive integer $n\ge 3.$ Importantly, these points are part of the set. But now let $\angle BAC$ and $\angle ACD$ be two consecutive angles of the convex hull. If $\angle BAC>\angle ACD,$ then the regular polygon with consecutive vertices $B,A,C$ contains $D$ in its interior (unless $\angle BAC=60^\circ,$ but then $\angle ACD<60^\circ$ but from above this is impossible). Thus this is impossible, and thus the convex hull is also equiangular, so it is regular. Now suppose the convex hull has $6$ or more sides. Then it is easy to see that two vertices that are $3$ sides apart must have a distance between them at least twice the side length. Then the circles centered at those vertices intersect at less than two points. But from (a) any two of the circles must intersect and not be tangent by considering the line through their centers, so this is impossible. Now we only have $3$ cases to check. Case 1: The convex hull is a triangle. Then any point in the triangle inverts to a point outside the triangle from any point, thus the set can only have $3$ points in this case. Case 2: The convex hull is a square. Let the square be $ABCD.$ The inverse of $C$ about the circle at $A$ is the center $O$ of the circle, which thus must be in the set. But now any point in $AOB$ is sent outside the square upon inversion at $A,$ and similarly for $BOC,COD,DOA.$ Thus the set must have $5$ points in this case. Case 3: The convex hull is a pentagon. Let the pentagon be $ABCDE.$ If $BD\cap CE=F,CE\cap DA=G,DA\cap EB=H,EB\cap AC=I,AC\cap BD=J$ then from (a) we get $F,G,H,I,J$ must also lie in the set. Now angle chasing shows that $F$ inverts to the center $O$ of the pentagon about the circle centered at $A.$ Additionally (a) and angle chasing show that the circle centered at $F$ passes through $C,D,H,I.$ Now let the circle at $A$ intersect the circle at $F$ at $X,Y.$ From (a) we get the circle at $O$ must pass through $X,Y.$ But the inversion at $A$ sends $H,I$ to $C,D$ and thus the circles at $A$ and $F$ are orthogonal, so $AXFY$ is cyclic. Also, Radical Axis theorem on the circles at $A,B,F$ shows that $O$ lies on $XY,$ and thus $OX\cdot OY=OF\cdot OA.$ But then this implies that $F$ inverts about the circle at $O$ to the reflection of $A$ over $O,$ which lies outside the original pentagon, contradiction. Thus we have shown that the maximum is $5$ points.

One remark on this problem is, if inversion power is allowed to be negative, the maximum number of point is $11.$ Construction is a regular 5-gon with 5 intersections and the center. The proof is quite complicated though.

Infinite Descent Pog For (a), let $OA_1A_2 \dots A_n$ be a regular $(n+1)$-gon. Invert about $O$ to map to $B_1B_2 \dots B_n$ on a line with radius $1$. Then we have that \[ B_kB_{k+i} \cdot B_kB_{n+1-i} = \frac{A_kA_{k+i} \cdot A_kA_{n+1-i}}{OA_k^2 \dot OA_{k+i} \cdot OA_{n+1-i}} = \frac{1}{OA_k^2} \]which gives the result. Now, for (b), we work on the following. Claim: If a line $AB$ only has two points $A,B \in S$ on it, then $S$ is fixed under reflection about the perpendicular bisector of $AB$. Proof. Let $(A)$ be the circle with center $A$ through $B$, define $(B)$ similarly. Inverting about $(A), (B), (A)$ is equivalent to reflection about said bisector (which is proven by considering the image of $\triangle PAB$). $\blacksquare$ Claim: If a finite set is closed under three distinct reflections and has finitely many points, then the three reflections share a fixed point. Proof. Let $O$ be the common fixed points of the first two reflections, else we are done. Else, the argument between the first two reflections is in $\mathbb{Q}\pi$ with denominator at least $\frac{1}{2}$. Then we can use the third operation after spamming the first two to get points arbitrarily far from $O$. $\blacksquare$ Then suppose $AB$ has no other points on it. Then reflecting about the circle though $A$ with center $B$, and the one through $B$ with center $A$, fixes $S$. Applying this operation a few times, this is equivalent to reflection about the perpendicular bisector of $AB$. As such, these $3$ lines concur at one point. Claim: Let $L$ be a line on the convex hull of $S$. Then $L$ only has two points. Proof. Let $P$ be the point in $S \setminus L$ closest to $L$. Then inverting about $P$ must fix all points on $L$, which implies that there are two points in $L$. $\blacksquare$ As such, the convex hull $\mathcal{S}$ of $S$ is a polygon $P_1P_2 \dots P_k$. By considering inversion about the points $P_i \in S$ on this hull, it follows that this hull is equilateral. By considering reflecting about the perpendicular bisector of $P_iP_{i+1}$, it follows that the polygon is cyclic and thus is regular. Claim: For $n \ge 5$, given a set $S$ of points $P_1P_2 \dots P_n$ contained in a regular $n$-gon such that inverting about $P_i$ with radius $P_iP_{i+1}$ fixes $S$, then there is a smaller polygon $Q_1Q_2 \dots Q_n$ and smaller set $S'$ in $S$ that satisfies this condition. Proof. The image of $P_{i+2}$ under inversion at $P_i$ is $Q_i \coloneq P_iP_{i+2} \cap P_{i+1}P_{i-1}$, and furthermore there are no other points in $S$ on $P_{i-1}P_{i+1}$ closer to $P_{i+1}$. Then since $Q_iP_i = Q_iP_{i+1}$, it follows that the inversion at $Q_i$ has radii $Q_iQ_{i+1}$ which finishes with considering the points in this polygon. $\blacksquare$ Repeating this infinitely implies $S$ is finite, contradiction. As such, we have that $n \le 4$. For $n = 4$ we get a configuration with square $ABCD$ and $E \coloneq AC \cap BD$ is also in $S$, this is maximal by inverting about $S$. For $n = 3$, we get an equilateral triangle. As such, $n = 5$ is maximal. Remark: Sylvester-Gallai serves as decent motivation for considering lines with only two points on them.