A:Yes. Bitwise EX-OR.

Do both have to hold simultaneously ?

@above yes.

Here's my approach using Group Theory but the approach maybe wrong because I know nothing much in depth about Group Theory. I would highly appreciate if anyone would point out the flaws. Ankoganit wrote: Is it possible to define an operation $\star$ on $\mathbb Z$ such that for any $a, b, c$ in $\mathbb Z, (a \star b) \star c = a \star (b \star c)$ holds; for any $x, y$ in $\mathbb Z, x \star x \star y = y \star x \star x=y$? Suppose $(\mathbb{Z},\star)$ forms a Group (Non-trivial) $\mathcal{G}$. Now we show that $\mathcal G$ can form a group by showing that conditions (a) and (b) does not contradict the criterias for forming a group. Condition (a) clearly does not contradict the criterias for forming a group. Now our claim is that $\mathcal{G}$ is abelian. From the second condition of (b) we get that $y\star x\star x=y\implies x\star x=1_{G}$ where $1_{G}$ is the Identity of $\mathcal{G}$. So, $x\star x=1_{G}\implies \mathcal{G}$ is a Self Inverse Group. Now Self Inverse Groups are Abelian. The proof goes as follows. Lemma:- Self Inverse Groups are Abelian. Let $\mathcal{G}=(G,\star)$. Let $a,b\in\mathcal{G}$. Hence, \begin{align*}(a\star a)\star(b\star b)=1_G\star 1_G=1_G &\implies a\star(a\star b)\star b=a\star(b\star a)\star b \\ &\implies (a\star a)\star(a\star b)\star(b\star b)\\ &=(a\star a)\star(b\star a)\star(b\star b)\\ &\implies (a\star b)\star 1_G=(b\star a)\star 1_G \\ &\implies a\star b=b\star a\\ \end{align*}Hence $\mathcal{G}$ is abelian. Now as $\mathcal{G}$ is abelain so the first condition of (b) implies $x\star x\star y=1_G\star y=y\star 1_G=y\implies$ every element has an identity. So none of the conditions (a) and (b) contradict $\mathcal{G}$ to be a group. So now it just suffices to prove that there exists a self inverse group $(\mathbb{Z},\star)$ (which I guess exists). If so, then we are done. $\blacksquare$

@above there does exists a group with all elements self inverse in $\mathbb Z$ but I avn't checked your proof tho.