Join CE. angle CEA = 90. Let CO = x. => AE = (a+x)/sqrt(2). => area of triangle AEO is (a)*((a+x)/(sqrt(2))*sin45/2 = (a)*(a+x)/4. Let angle CAD = T. In triangle ACF, CF = (a+x)sinT. In cyclic quad DOCF, angle CFO = angle CDO = 90 - angle DCO = 90 - angle OAB = 90 - 45 = 45. Hence area of triangle COF is (b)*(a+x)sinT*sin45/2 => ar(COF)=(b)*(a+x)sinT/(sqrt(2)*2). Now, in triangle AOF, by sine rule, b/sinT=a/sinOFD=a/sinOCD=a/sinOAB=a/sin45=a*sqrt(2). =>sinT=b/(a*sqrt(2)). Hence the ratio of areas of AOE and COF turns out to be (a*a)/(b*b).

$\angle OAE=45=\angle ODC=\angle OFC$. Since $AECF$ is cyclic, $E,O,F$ are collinear, so $\angle AOE=\angle FOC$. Thus $\triangle COF\sim\triangle EOA$. It follows that $\frac{[AOE]}{[COF]}=\frac{a^2}{b^2}$.