We want to find an $n$ such that the average of the first $n$ integers is closest to $\frac{163}{4}$. The average of the set will be $\frac{n+1}{2}=\frac{163}{4}$ $n+1=81.5$ $n=80.5$ Thus, trying $n=81$, we find that the sum of all of the integers is $\frac{(81)(82)}{2}=3321$. Removing one integer will obtain an average of $\frac{163}{4}$, so the sum of all the elements in the set with one removed will be $\frac{163}{4}\time 80=3260$. Thus, the removed number is $3321-3260=\fbox{61}$.