The problem tells us to find all numbers $n$ such that $$ab \equiv \pm 1 \pmod{n} \implies a \equiv \pm 1 \pmod{n}$$We effectively remove the 'at least one of $a$ and $b$' condition since if $a \equiv \pm 1 \pmod{n}$ and $ab \equiv \pm 1 \pmod{n}$ then naturally if $b \not \equiv \pm 1 \pmod{n}$ then $ab \equiv \pm b \pmod{n} \not \equiv \pm 1 \pmod{n}$. The condition means that there are no integers $a$ with $a \not \equiv \pm 1 \pmod{n}$ such that $\gcd(a, n) = 1$ by simple properties of modular inverses and the problem is equivalent to finding all $n$ with $\phi(n) \le 2$. It is simple to show that this set is $\{1, 2, 3, 4, 6\}$.