A common divisor of $A,B,C$ is also a divisor for $D=2A-B$, $E=3A-C$, $F=5E-7D$, $G=5D-E$, $H=18A-2F-3E$, $I=nG-mF$ and $126=18nI-5H+11F=2\cdot 3^2 \cdot 7$. Since $2$ and $3$ do not divide $A,B$ and $C$, then $d=7$. It follows that $(m,n)$ is equal to $(7a+2,7b+3)$ or $(7c+5,7d+4)$.

Where did the $D,E,F,G,H,I$ came from, and how is "$126=18nI-5H+11F=2*3^2*7$" true? Also, how are the solutions derived by only knowing $d=7$? (I literally don't understand the whole solution) Sorry, but may someone explain?

I do not understand it either :/

It is however easy to understand. The quantities $D,E,F,G,H,I$ are computed from $A,B,C$ (that is where they come from). So the conclusion is a common divisor $d$ must divide $126 = 2\cdot 3^2\cdot 7$. But $A+B+C = 6(m^2+mn+n^2) + 5$, so $d$ is coprime with $6$, and so it can only be $d=7$

H=18A-2F-3G, not 3E.

mousavi wrote: A common divisor of $A,B,C$ is also a divisor for $D=2A-B$, $E=3A-C$, $F=5E-7D$, $G=5D-E$, $H=18A-2F-3E$, $I=nG-mF$ and $126=18nI-5H+11F=2\cdot 3^2 \cdot 7$. Since $2$ and $3$ do not divide $A,B$ and $C$, then $d=7$. It follows that $(m,n)$ is equal to $(7a+2,7b+3)$ or $(7c+5,7d+4)$. This is the solution proposed by the creator.

I propose a more direct calculation to obtain $d=7.$ We begin like mousavi did, but make changes near the end \begin{align*} D &= 2A-B = mn+5m^2+2, \\ E &= 3A-C = 5mn+7m^2+5, \\ F &= 5E-7D = 18mn+18. \end{align*}Now, we would like to manipulate $A,B,C$ such that we can cancel $m^2$ and $n^2$ to make use of $F.$ Nothing that the ratio of coefficient of $m^2$ to $n^2$ is $2:3$ in $C,$ we try to manipulate $A$ and $B$ to match such ratio. We can do this by calculating $A+7B=15n^2+23mn+10m^2+16$ so $$A+7B-5C=18mn+11.$$Finally, $\boxed{F-(18mn+11)=7.}$

I got common divisor d=7 by subtracting multiples of A, B, and C together, but how do I continue? I don't know if there is a faster way than literally testing all 49 cases of the residues mod 7 of each of m and n. Anyways, here is some motivation for the constant 126 as a sum of multiples of A, B, and C: It may seem like the first post's answer to this problem to get a common divisor was way too spontaneous, like "I get WHY that works, but how would I have found that?" Well, let us represent this by $d=aA+bB+cC=an^2+2amn+3am^2+2a+2bn^2+3bmn+bm^2+2b+3cn^2+cmn+2cm^2+c$. Then to get rid of all the multiples of n^2, mn, m^2, we just need the coefficients to sum to 0. So we have $a+2b+3c=0$, $2a+3b+c=0$, and $3a+b+2c=0$. I leave others to solve this.

chfrn wrote: I propose a more direct calculation to obtain $d=7.$ We begin like mousavi did, but make changes near the end \begin{align*} D &= 2A-B = mn+5m^2+2, \\ E &= 3A-C = 5mn+7m^2+5, \\ F &= 5E-7D = 18mn+18. \end{align*}Now, we would like to manipulate $A,B,C$ such that we can cancel $m^2$ and $n^2$ to make use of $F.$ Nothing that the ratio of coefficient of $m^2$ to $n^2$ is $2:3$ in $C,$ we try to manipulate $A$ and $B$ to match such ratio. We can do this by calculating $A+7B=15n^2+23mn+10m^2+16$ so $$A+7B-5C=18mn+11.$$Finally, $\boxed{F-(18mn+11)=7.}$ F=5E-7D=A+7B-5C so F-A-7B+5C=0