[Moderator edit: The below solution is copied from http://www.kalva.demon.co.uk/short/soln/sh00g5.html .] We use the triangle $ABA'$ to find $x$. Let $O$ be the center of the circle and $R$ its radius. Then angle $OBA' = 90o$ and ∠A'OB = 1/2 ∠BOC = ∠A, so BA' = R tan A. Also ∠AOB = 2∠C, so AB = 2R sin C. ∠A'BA = ∠A, so ∠A'BA = ∠A + ∠B. Hence ∠AA'B = 180o - ∠A - ∠B - x = ∠C - x. Now from the sine rule, AB/sin AA'B = A'B/sin x. Hence sin(∠C - x) = 2 sin ∠C sin x cot ∠A. So sin ∠C cos x - cos ∠C sin x = 2 sin ∠C sin x cot ∠A. Hence cot x = cot ∠C + 2 cot ∠A. Applying the sine rule to BMA and BMD gives sin y/sin x = MA/MB = MD/MB = sin MBD/sin MDB = sin(B - y)/sin(B + x), or sin y sin(B + x) = sin x sin(B - y). Expanding and dividing through by sin x sin y sin B gives cot x + cot B = cot y - cot B, or cot y = 2 cot B + cot x = 2 cot A + 2 cot B + cot C. This is symmetric between A and B, so cot ABM = cot BAN. But cot is strictly decreasing over 0o to 180o, so ∠ABM = ∠BAN. We want to maximise ABM or minimise cot ABM and hence minimise 2 cot A + 2 cot B + cot C. But 2 cot A + 2 cot B = 2 sin(A+B)/(sin A sin B) = 4 sin C/(cos(A - B) - cos(A + B) ) = 4 sin C/(cos(A - B) + cos C) ≥ 4 sin C/(1 + cos C) with equality iff A = B. Thus we must take A = B. We then want to minimise 4 sin C/(1 + cos C) + cot C = 4 (2 sin k cos k)/(2 cos2k) + (1 - tan2k)/(tan k + tan k), where k = C/2, and hence to minimise 7/2 tan k + 1/(2 tan k). That is achieved by taking tan k = 1/√7 (eg complete the square), and hence sin C = sin 2k = 2 tan k/(1 + tan2k) = (√7)/4. Since A = B, we have AC = BC and 1 = AB = 2 AC sin k = 2 AC 1/√8. Hence AC = √2.

The < ABM = < BAN part cries for a synthetic proof. And indeed, there is one, but it is quite hard. I have just uploaded it on my website: see the paper "Three properties of the symmedian point", Theorem 6 a). Darij

I don't know if the part (a) of the problem ( $ \angle ABM = \angle BAM$ ) cries for a synthetic proof yet, but an elementary one, it has already been posted in the topic Angle with tangential circle, of discredit. Kostas Vittas.

Here's a quick solution with lengths: Let $A_1,B_1,C_1$ be the midpoints of $BC,CA,AB$. Note that $Q$ is the foot of the $A$-symmedian in $\triangle AB_1C_1$. Now let $D=BQ\cap A_1B_1$; similarly define $E=AS\cap A_1B_1$. It follows that $B_1D:BC_1=B_1Q:C_1Q=b^2:c^2\implies B_1D=\dfrac{b^2}{2c}$. It follows from SAS that $\triangle DB_1A\sim \triangle CAB$. Similarly we get $A_1E=\dfrac{a^2}{2c}$ and $\triangle EA_1B\sim \triangle CBA$, so $\angle D=\angle E = \angle C$, so $ABED$ is an isosceles trapezoid, implying $\angle DBA=\angle EAB\implies \angle QBA=\angle SAB$ as desired. For the second part, let $BD,AE$ meet at $X$ and let $N$ be the projection from $C_1$ onto $A_1B_1$. Then $C_1X:XN=AB:DE=\dfrac{2c^2}{a^2+b^2+c^2}$ using the formulas for $B_1D,A_1E$ from earlier. Now let's fix $A,B$ and vary $C$ on a line parallel to $AB$ so $N$ is fixed. Then to maximize $\angle XBA$ clearly we want to maximize the ratio $C_1X:XN$, which amounts to minimizing $a^2+b^2$. As $C$ varies clearly this happens when $CA=CB$, implying $a=b$. Now let $h_c$ be the length of the altitude from $C$ so that $C_1X=\dfrac{MX}{MX+XN}\cdot \dfrac{h_c}{2}=\dfrac{2c^2}{a^2+b^2+3c^2}\cdot \dfrac{h_c}{2}=\dfrac{c^2\sqrt{4a^2-c^2}}{4a^2+6c^2}$. Now it follows that $\tan \angle XBA=\dfrac{\sqrt{c^2(4a^2-c^2)}}{2a^2+3c^2}$; to maximize this angle we should maximize the tangent, and by AM-GM we have $\dfrac{\sqrt{7c^2(4a^2-c^2)}}{\sqrt{7}(2a^2+3c^2)}\le \dfrac{2a^2+3c^2}{\sqrt{7}(2a^2+3c^2)}=\dfrac{1}{\sqrt{7}}$ with equality when $7c^2=4a^2-c^2\implies a:b:c=\sqrt{2}:\sqrt{2}:1$.

I'll prove the first part of the problem. (Very similar to #5) Let $M,N$ - midpoints of $BC,AC$. And $AM,BN$ meet $(ABC)$ at $X,Y$. 1) $ABPQ\sim AXCN$ and hence $\angle ABQ=\angle AXN$. Similarly $\angle BAS=\angle BYN$. 2) Note that centroid of $ABC$ is incimilicenter of $(ABC)$ and $(MNC)$ and so it lays on their radical axis. Hence by Radical Axis Theorem $MNXY$ is cyclic. Combine 1) and 2) for the final result.

Let $BQ$ intersect $AC$ at $D$, $AS$ intersect $AB$ at $E$. Let $F$ be on $AC$ and $G$ be on $AB$ such that $PF\parallel BD$ and $RG\parallel AE$. By the properties of a symmedian, we have $BP:PC=AB^2:AC^2$ and $AR:RC=BA^2:BC^2$. Also note that $AD=DF$ and $BE=EG$. Thus, we have \[\frac{AD}{DC}=\frac{AD}{DF+FC}=\frac{AD}{DF+DF\frac{CP}{BP}}=\frac{AD}{AD\left(1+\frac{AC^2}{AB^2}\right)}=\frac{AB^2}{AB^2+AC^2}\]and similarly, $\tfrac{BE}{EC}=\frac{AB^2}{AB^2+BC^2}$. Let $a=BC$, $b=CA$, $c=AB$. Let $\alpha=\angle BAC$, $\beta=\angle ABC$, and $\gamma=\angle CAB$. By the Law of Sines, \[\frac{\sin(\angle BAE)}{\sin(\alpha)\cos(\angle BAE)-\cos(\alpha)\sin(\angle BAE)}=\frac{\sin(\angle BAE)}{\sin(\angle CAE)}=\frac{BE}{CE}\cdot \frac{AC}{AB}=\frac{bc}{c^2+a^2}\]When we take the reciprocal, we get \[\sin(\alpha)\cot(\angle BAE)-\cos(\alpha)=\frac{c}{b}+\frac{a^2}{bc}\]However, we know that $\cos(\alpha)=\tfrac{b^2+c^2-a^2}{2bc}$ so \[\cot(\angle BAE)=\frac{3c^2+a^2+b^2}{2bc\sin(\alpha)}\]Similarly, \[\cot(\angle ABD)=\frac{3c^2+a^2+b^2}{2ac\sin(\beta)}\]and since $b\sin(\alpha)=a\sin(\beta)$ it is clear that $\cot(\angle BAE)=\cot(\angle ABD)$. Since $\cot$ is injective in $(0,\pi)$, $\angle BAE=\angle ABD$ as desired. Now, to maximum this number, we need to minimize \[\cot(\angle BAE)=\frac{3c^2+a^2+b^2}{4[ABC]}=\frac{3c^2+a^2+b^2}{\sqrt{(a+b+c)(a+b-c)(a-b+c)(-a+b+c)}}\]We claim that the minimum is $\sqrt{7}$. Indeed, note that $\left(1,1,\tfrac{\sqrt{2}}{2}\right)$ attains that minimum. It remains to show that $3c^2+a^2+b^2\ge \sqrt{7(a+b+c)(a+b-c)(a-b+c)(-a+b+c)}$. We can square both sides to get \[9c^4+a^4+b^4+6a^2c^2+6b^2c^2+2a^2b^2\ge -7a^4+14a^2b^2+14a^2c^2-7b^4+14b^2c^2-7c^4\]Rearranged and divided by $4$, this is \[4c^4+2a^4+2b^4-2a^2c^2-2b^2c^2-3a^2b^2\ge 0\]Which is true because \[\frac{1}{2}(2c^2-a^2)^2+\frac{1}{2}(2c^2-b^2)^2+\frac{3}{2}(a^2-b^2)^2\ge 0\]And the equality condition is that $a:b:c=2:2:\sqrt{2}$.

Here's a quick and easy proof of part (a) with barycentric coordinates: Let $\omega$ be the circumcircle of $\triangle ABC$. Suppose $W = BQ \cap AS$. We will show that $W$ lies on the perpendicular bisector of $AB$. It is well known (and easy to show) that $T= (-a^2:b^2:c^2), \ U = (a^2:-b^2:c^2)$. Therefore $P = (0:b^2:c^2), \ R = (a^2:0:c^2)$. Thus, the midpoint of $AP$ is given by $Q = (A+P)/2 = (b^2+c^2:b^2:c^2)$, and similarly $S = (a^2: a^2+c^2: c^2)$. Finally, $W = BQ \cap AS$ is immediately seen to be of the form $(b^2+c^2: * : c^2)$ as well as $(*: a^2+c^2: c^2)$. Hence, $W = (b^2+c^2: a^2+c^2: c^2)$. It remains to show that $W$ lies on the perpendicular bisector of $AB$. The equation of the perpendicular bisector of $AB$ is well known to be $(a^2 - b^2)z + c^2(x - y) = 0$. And this can clearly be seen to be satisfied by $W$. $\square$

Posting this due to a request of a friend of mine. This is the longest length bash I have done in a while... Let $\angle ABQ=\theta $, $\angle BAS=\alpha $. In the first part of the problem we desire $\alpha=\theta$ First we use two ratio lemmas in $\triangle ASP$ and $\triangle BAQ$, we get: $$\frac{sin(\alpha)}{sin(A)}.\frac{AS}{AR}=\frac{BS}{BR}=\frac{1}{2}$$$$\frac{sin(\theta)}{sin(B)}.\frac{BQ}{BP}=\frac{AQ}{AP}=\frac{1}{2}$$Also note that clearly $sin(\theta)=sin(\alpha)$ clearly suffices since the to lines $AS,BQ$ intersect inside the triangle; Thus we have $sin(\theta)=sin(\alpha)$ $\iff$ $$\frac{sin(A)}{sin(B)}=\frac{AS}{AR}.\frac{BP}{BQ}$$Also note that we have the lengths of $AR,BP$ and they are $\frac{bc^2}{a^2+c^2},\frac{ac^2}{b^2+c^2}$ respectively.(Since $BU,AT$ are symmedians).Now we use $\frac{sin(A)}{sin(B)}=\frac{a}{b}$ and the above to get: $$\frac{sin(A)}{sin(B)}=\frac{AS}{\frac{bc^2}{a^2+c^2}}.\frac{\frac{ac^2}{b^2+c^2}}{BQ}$$$$\iff$$$$\frac{AS}{BQ}=\frac{b^2+c^2}{a^2+c^2}$$ Squaring both sides and using the length of the median in a triangle: $$\frac{AS^2}{BQ^2}=\frac{(b^2+c^2)^2}{(a^2+c^2)^2}$$$$\iff$$$$\frac{\frac{c^2}{2}+\frac{AR^2}{2}-\frac{BR^2}{4}}{\frac{c^2}{2}+\frac{BP^2}{2}-\frac{AP^2}{4}}=\frac{(b^2+c^2)^2}{(a^2+c^2)^2}$$Then using Stewart's theorem to get the lengths of $BR^2,AP^2$ and also as previously mentioned, $AR=\frac{bc^2}{a^2+c^2}, BP=\frac{ac^2}{b^2+c^2}$ , we arrive at the following: $$\frac{\frac{c^2}{2}+\frac{b^2c^4}{2(a^2+c^2)^2}-\frac{a^2c^2}{2(a^2+c^2)}+\frac{a^2b^2c^2}{4(a^2+c^2)^2}}{\frac{c^2}{2}+\frac{a^2c^4}{2(b^2+c^2)^2}-\frac{b^2c^2}{2(b^2+c^2)}+\frac{a^2b^2c^2}{4(b^2+c^2)^2}}=\frac{(b^2+c^2)^2}{(a^2+c^2)^2}$$$$\iff$$after cross multiplying and multiplying the whole equation by $4$, we get that, $$L.H.S=2c^2(c^2+a^2)^2+2b^{2}c^{4}-2a^{2}c^{2}(a^2+c^2)+a^2b^2c^2=2c^6+2a^2c^4+2b^2c^4+a^2b^2c^2$$which is symmetric w.r.t $a,b$ and we are done.