From BrahmaGupta's Inequality we see if the area of the quadrilateral is $S$, then: \[S\le \sqrt{(11-3)(11-5)(11-6)(11-8)}=12\sqrt{5}=3\sqrt{80}<27<9\pi\] And since the area of the circle is less than $S$, the radius must be less than $3$ as desired.

The fact that the area is $S = \sqrt{(s-a)(s-b)(s-c)(s-d) - abcd\cos^2 \frac {\angle A + \angle C} {2}} \leq $ $\sqrt{(s-a)(s-b)(s-c)(s-d)}$ is also known as Bretschneider's formula. That this kills the problem may come as a surprise to the setters; both official solutions are obviously more contrived.

But even Ptolemy's Inequality kills it. If $X, Y$ are the lengths of the diagonals and $\theta$ the angle between them, then: \[S=\frac{XY\sin\theta}{2}\le \frac{XY}{2}\le \frac{3\cdot 8+5\cdot 6}{2}=27<9\pi,\] which again gives the result. Edit: I read "contained in" as "inscribed", now I see this isn't so.

Well, at least this approach was prohibited by the problem setters. The lengths of successive sides are $3,6,5,8$, so the formula involving Ptolemy's inequality is \[S=\frac{XY\sin\theta}{2}\le \frac{XY}{2}\le \frac{3\cdot 5+6\cdot 8}{2}=\frac {63} {2} >9\pi.\]

Ah yea. But then we can still let the angles between sides $3, 8$ and $5, 6$ be $\alpha, \beta$ respectively, then: \[S=\frac{3\cdot 8\cdot \sin\alpha+5\cdot 6\cdot \sin\beta}{2}\le \frac{3\cdot 8+5\cdot 6}{2}=27<9\pi.\] To get the result again!